I am thinking of setting up a redundant server that runs a disaster-recovery trading bot in case the main bot in a NYC datacenter can not function due to IT failures caused by weather or other unpredictable events(e.g. AWS servers were down at least once during regular trading hours in the past 12 months). Is there any advantage of placing the redundant server in Chicago? I might get into trading options in the future. Thanks!
In terms of AWS, it is commonly said amongst developers not to run anything in production on US-EAST-1 region due to the frequency of outages in that region. A possible benefit of placing your redundant server in Chicago is to diversify your location risk from East Coast region outages like these: https://gothamist.com/news/internet-outages-frustrate-east-coast-verizon-blames-cut-cable-brooklyn https://www.adn.com/nation-world/20...t-has-caused-large-parts-of-the-web-to-crash/ We specialize in Chicago servers for running critical trading applications 24/7. The latency from our Chicago servers to NYC is ~22ms. Let me know if you're interested in testing and I can set you up with a free test server.
I think you will get most bang for your buck and higher uptime first by getting a better, dedicated hosting provider within NJ rather than getting a backup. Most vendors were still up at NY4 even when Hurricane Sandy knocked out most of Manhattan. And the WAN network between your main and backup is likely more unreliable than the moving pieces within a data center. And your uptime will be nearly on parity with the matching engine if you're in the same facility - it's arguably better that you have only 99% uptime but your downtime is in sync with your broker ISV's order execution gateway or market order gateway, than 99.999% uptime that is out of sync with gateways that you need to reach. The problem with backups and redundancy is that it sounds good in principle, but they introduce complexity that can be worse if you don't spend the effort on setting up a good recovery process. And recovery is a lot more challenging than backup. For instance: - How does your backup application know the exact residual position and orders at the time your main one failed? Drop copy? Asynchronous messaging between them? - What if your main application is actually still running but just unreachable?
Thank you for your advice. My strategies are not sensitive to occasional interruptions for 1-2 minutes. If heart beat of the main application can not be detected, the backup application can send a command to the shutdown the machine running the main application via iDrac. Would this address the 2nd issue you mentioned? https://www.hungred.com/how-to/command-line-start-stop-reboot-remote-server-idrac-racadm/
Your strategy seems viable, especially if the backup application itself doesn't make any order actions. But I still think a better hosting provider will get you higher uptime. For example, what if the main application is just actually running but merely unreachable outside of the data center because the VPN is down or the OOB management network is down? Then you still won't be able to reach it via iDRAC. Single bare metal servers can stay up and running for years, but WAN links drop frequently.