September 20, 2016
For those of you of a certain age, you may remember that one of the burning questions in our salad days was who was the loudest rock and roll band in the world—note the past tense because does anyone go to a Taylor Swift concert to hear “Shake it Off” at a decibel level akin to a jet engine? No need to answer that one. But back in the day you couldn’t attend a performance by your favorite band without your ears ringing for the next two days, and we thought that was a good thing. And, although we didn’t think listening to Steven Tyler at volumes guaranteed to blow out your speakers—also considered a good thing—would have no adverse effects, it turns out that we might have been wrong in our assumption. As the Romanian customers serviced by ING Bank’s regional data center recently found out, noise levels that approximate Jimmy Page power chords can have a negative impact on hard drive performance.
Someone once said that, “The road to hell is paved with good intentions” and this incident seems to qualify. Because no one likes a fire in their data center, the proactive ING technicians decided it might be a good idea to test their Inergen fire suppression system to ensure that it was capable of forestalling a server conflagration, only to experience what, euphemistically, might be described as unintended consequences.
When the system was turned on the resulting sound damaged a large number of the facility’s hard drives. Apparently not even ear plugs could have prevented the resulting service outage that left over one million of their customers unable to use the ATM network, credit cards or even check their balances via ING’s home banking system. According to bank officials, the noise level of the escaping gas reached 130 decibels—we’re talking AC/DC territory here—with the corresponding vibrations “shell shocking” surrounding hard drives. The bank even put out an official statement to explain the event. Interestingly, no one who was unable to withdraw a few Romanian Lau for 10 hours was available for comment.
Since the event, a number of groups, individuals and publications have come forward to advise us all that the high decibel levels arising from inert gas extinguishing systems, and their impact on hard drives, is a known problem, prompting ING data center technicians to respond with a collective, “now you tell me”. As a member of the “louder is better” generation I don’t think we should judge them too harshly. When your parents warned you not to sit so close to the TV because it would hurt your eyes or don’t listen to Zeppelin IV with your headphones on and the volume turned up to ten, did you listen? Some lessons in life can only be learned through experience. Those of you with children probably understand this better than some of the other readers of this blog. That is what is known as karma, and a call to your own parents for guidance would only prompt hysterical laughter on the other end of the line so you’re going to be on your own for a few more years. From a practical perspective, I think the folks at ING have learned their lesson. In the world of data centers, unlike rock and roll, you never want to turn things up to 11.