Our company just ran a DR test, where they pretended the datacenter had been destroyed and that they had to activate the one in Philadelphia. They spent many thousands to fly ten or so people up there. And when they got there? They couldn’t do it. Why?

Nobody checked beforehand if the backups were valid and would restore. Turns out, they can’t. A one hour test could have determined that beforehand. And one of my team’s managers has been bugging them since September to do so before the rest got scheduled.

So, travel money for the team? Lets say $20,000. Complete loss of productivity for the ten people in Philly and the other 10-20 here at the office that were 100% dedicated to the project? Two or three days worth? Dunno if I can calculate it.

The CIO is said to be livid. We’ve been warned to physically avoid being in the vicinity of his office. Fortunately I wasn’t involved in DR at all, my systems aren’t deemed critical enough. But damn.