A Business Destroyed in One Night

It's been a long day and dinner with the family was especially nice this evening. I had a conversation with my sons about what we are planning for this fall; campouts, a trip to the trout lodge, maybe taking them on their first pheasant hunt as soon as the red leaves start their descending trip from the high oaks to the forest floor. We tucked the kids into their beds, prayed our nightly prayers and then my wife and I enjoyed dusk and twilight on our deck with a great bottle of wine sent to us from a friend in Southern California... altogether a perfect night, especially for a Thursday. I rolled over in bed and tucked my hands beneath the pillow, ready for an early Friday and a welcome weekend.

I felt as though I had barely closed my eyes when I heard the phone ring. It took me a couple minutes to remove it from my dream and place the sound into reality. I blinked, rubbing the sleep from my eyes and looked to the clock on my dresser; 3:27 AM. Who could be calling me? I jumped out of bed right as the call went to voicemail. As I reached for the phone, I noted that I had already missed 4 calls. Tonight? As I unlocked the BlackBerry, the phone vibrated and rang again... my IT Director? ...No. No, no. Surely not.

I answered. In a toiled voice, almost a ringing in my sleep-filled head, I proceeded to listen to my IT Director tell me that, almost impossibly, around 10 PM, the building had experienced a power spike which had blown the breaker on the HVAC, killing the server room's AC. Fine, the monitor must have notified... no? Monitor had failed on Wednesday morning. The domain controller finally sent a message to his email stating disk failure... only after the RAID had failed and the ambient temperature had risen to 92F. Complete failure. The AC coil had thawed and had leaked water all over the floor, shorting out the battery backup. Complete disaster. Tapes? Haven't been switched out in a week and a half...

The icing on top of all of this? Our largest general contractor has a hospital bid going out in the morning... had, a bid going out... this morning. What about litigation? What if they sue? How do I get the data back? Could this be enough to ruin my business?

Now what?

This isn't my story. Hopefully its not yours either. But without proper backup, both on site and off site, its going to be, sooner or later. Failures happen, machines fail, the world aligns against you and your success. You must be prepared. For complete failure and complete devastation.

I have always equated backup and recovery solutions for critical data to an old adage I heard when shopping for riding gear for my sport bikes. I bought the jacket with armor, the pants, the riding boots, Kevlar gloves, back protector and I was looking to replace my helmet. Helmets range from $100 for a good full face, off-brand to over $1000 for a nice Arai. When I asked a friend what I should spend on a helmet, he said, simply: "What's your head worth?" Seems simple. It's worth everything. I can't afford to lose it, I can't afford to damage it. I had a $10,000 bike but really, the bike was a fraction of what it truly took to "ride". I spent another couple thousand on the appropriate gear to insure its (and my) performance.

Hosting your own data center is no different. If you're going to ride, you'd better have the gear necessary to insure its performance. Tape backup is not enough. RAID 5 is not enough. You could have 30 snapshots of data backed up up on, real time on a SAN and on tapes but if your building burns down at midnight, none of the data inside matters anymore, no matter the preparation.

Preparing yourself to be an 'Information Manager' entails more than just having the gear, it means backing up, first to live hardware, then to tapes, then off site. It's a multi-tiered approach that has no excessive extents. Plan for the worst to happen. All your monitoring equipment fails, the building burns down and your IT team forgot to rotate tapes... what do you do? In the perfect situation, you pull your data from your off site backup (RMT's Data Center), and you restore to your backup equipment located in your storage unit or alternate production facility (another regional associate or partner).

The time to plan is now because someday soon it may be your Thursday night. Sleep well.

All the best,
Tanner