Our Own Worst Enemy
Winston Churchill once described United States’ Secretary of State John Foster Dulles as, “The only bull I know who carries his own china shop around with him”. While Dulles undoubtedly objected, isn’t it really a pretty apt descriptor for all of us? After all, we are only human and, as a result, we make mistakes. Usually not on purpose mind you, but sometimes even our best efforts can result in some degree of mayhem. Nowhere is this truer than in the data center. Both anecdotally and empirically, it is well understood that the leading cause of data center outages is human error. No one wants to be the one to bring an entire facility to its knees, but obviously the most common explanation in a service interruption post-mortem is usually who and not why. Since we know that we are our own worst data center enemies, doesn’t it seem like someone would start designing these things to help reduce the margin for “human error”?
While no design may be foolproof, aren’t there at least a few things that could be done to reduce the likelihood for those “oh s**t” moments that can ruin everyone’s day in the blink of an eye? How about providing front access to equipment to make them easier it to maintain? Having to service a CRAH should not require the average technician to possess the flexibility of a member of the US woman’s gymnastic team. Those girls are all about 12 years old and four feet tall, the average data center professional is…well, a little older, bigger and vaguely remembers the day he could touch his toes. Shouldn’t front access be a standard feature for reliability certainly, but also out of pure human compassion?
Data centers by definition are complex environments. Finding and correcting problems within a jungle of conduit that would have forced Stanley to leave Dr. Livingstone to fend for himself is not the best way to ensure efficient maintenance and the quick resolution of issues when they arise. In an environment where the only calls you get are to tell you something is wrong, couldn’t we help these guys out just a bit? How about color coding and labeling the conduit? Not only would it make navigating through the facility a lot easier it certainly looks pretty cool. Personally, I think anytime you can marry ergonomics and visual appeal, you’ve got a winning combination.
It might also be a nice touch for data center providers to provide detailed written Operating Procedures and Sequence of Operations and settings to the folks who will be supporting the facility before they turn it over. Although you’d think that this would be a given, most data center customers get the equivalent of a couple of paper clipped pages documenting their new facility along with the keys to the joint upon turnover. On the job training and trial and error are both effective tools for learning in the right environment, unfortunately a new data center isn’t one of them. Let’s face it, when your new car comes with more documentation that your multi-million dollar data center, you’ve got a problem.
I guess the fundamental question here is why aren’t data centers designed with their users in mind? If human error is the biggest obstacle to data center reliability then build facilities that minimize that potential. In the near future more customer oriented, ergonomic features that reduce the possibility for human error will undoubtedly become standard requirements, if not for merely for the sake of reliability but to help save us from ourselves.