One of the services that we provide to our clients is Disaster Recovery and High Availability design and planning and also implementation of those plans. A common question that always come in almost all of these meetings is about site disaster and many of the clients chose to by pass as not a valid concern since they are hosting with the top names in the industry. These companies do that in order to save costs since a similar infrastructure is typically required at the remote site as well which will take up the load in the event of the site failure.
That maybe ok for some companies based on their SLAs (Service Level Agreements) that they have with those hosting companies but if your application is a mission critical application and also you have declared an uptime for your applications in your SLAs to your clients, you better make sure that you have a right plan in place to take care of any site failures as well. In addition, another thing to understand is that have a proper plan for a site failure allows you to also have flexibility in your maintenance routines without much interruption of service.
Anyways, so why do we point this out today? Because we just heard from a client company that their site was down and it was down because a transformer outside the data center exploded because of inclement weather conditions and that area had lost electricity. The data center’s backup generators made sure that the web-sites and the applications were up and running but were not able to connect back to the internet due to loss of electricity in a radius of over 5-8 miles – it took nearly 10 hours for them to get the problem fixed. And as per the hosting company, this scenario did not qualify in the category of the site failure since the data center was up and running 🙂 . Now, this client wasn’t hosting with any of the well known names in the industry who typically have a very robust backup plan for such events but I just wanted to point out another reason to be paranoid about when you are in the ASP model and are running your business online and want a 24×7 operation during the entire year. Plan your DR and HA accordingly and ensure that you have tested those plans on a routine basis.