Don’t Touch the Red Button
Does anyone remember the classic Daffy Duck cartoon where he warns Porky Pig to never touch “the red button”, while our favorite stuttering porker succumbs to his curiosity and presses it anyway and mayhem ultimately ensues? If you do great; if you don’t, that’s too bad because it was hilarious. By now, many of you are probably wondering what this reference to a classic Warner Brothers cartoon means to you. Well if you’re a web hosting, cloud services or hosting provider the answer is a lot since an outage in your data center can also unleash a variety of unpleasant customer related responses. Remember sports fans, the SLA you have really doesn’t guarantee your operations will never go down, but it does specify what it’s going to cost you when they do. This is important due to the fact that a recent research report from the 451 Group indicates that unplanned service outages happen a little more frequently than we’d like to think.
The report documents the findings of a recent survey of data center operators conducted by Emerson Network Power and the Ponemom Institute that found that 90% had experienced at least one unplanned outage in the past two years. For those of you who find this fact disturbing enough, let me paraphrase the old line from those late night infomercials when I say, “ But wait, there’s more”. 73% of those who reported having an outage went on to say that they had fallen victim to 1 to 2 complete facility shutdowns during the period. Fortunately, just like the unexpected ability of the fabulous Ginzu knife to cut through a beer can, the report’s authors indicate that there is a positive aspect to all of this.
As many of you probably suspected based on your own experiences, the greatest enemy of data center uptime is the folks who actually operate the facilities. In other words, we have met the enemy and it is us. This doesn’t mean that the average data center is staffed with personnel who make the Three Stooges appear to be the epitome of competence, but it does mean that there are very specific things that you should look for in a provider.
The first element that the authors indicate that a potential provider should demonstrate is an effective program of technician training coupled with comprehensive monitoring and management capability. This strategy addresses some of the most common reasons for service interruption including UPS battery failure and exceeding UPS capacity. A critical element of this strategic element is the ability to view what’s going on with a BMS, and preferably, a DCIM overlay. As the old management adage goes, you can’t fix what you can’t see. Batteries that aren’t monitored, and UPS that you have to be at the panel to see the load are pretty much worthless in our outage prevention fight.
Perhaps the most notable finding of the survey was that although almost every data center will experience an outage at some point in time, the design and operation of the facility is the most important element in reducing the frequency and duration of their occurrence. Data centers adhering to the Uptime Institute’s Tier III standard for design and maintenance and operations best practices experienced 1.45 outages in the surveyed period as opposed to 2.87 outages for non-compliant facilities. In other words, doing the due diligence to ensure that your provider actually delivers you a Tier III certified data center is the most effective strategy that you can implement to ensure that any outages in your data center are infrequent and negligible in their customer impact.
As any service provider knows, and the 451 Group reinforces, unplanned outages are the bane of any operator’s existence. The key element that the study also brings to the fore is that the “human error” factor that is the biggest contributor to these issues extends to more than a lack of training, but a lack of doing the proper analysis of the solutions offered by potential vendors during the selection process. Ensuring that your next data center provider can deliver a reliable operational environment is the best way to avoid providing customers with the opportunity to give you their best Porky imitation when they tell you, “TH-TH-TH-TH-That’s all folks”.