There’s an old adage that shows up in many forms:
Smart people learn from their own mistakes; really smart people learn from the mistakes of others.
I’d expand that to include learning from one’s successes and the successes of others, as I’ve endeavored to teach my children many times over.
I’ve been in this business over 25 years and worked at Apple for a few years before that. Over that time, I’ve seen a lot of interesting customer experiences, good and bad, with some providing major life lessons the hard way. Most of these are simple in nature but reinforce what many of us know intuitively. The stories I want to share today relate to backing up.
I have a good friend who does great work with folks with disabilities. That is her life and passion. Early on, her department dedicated a computer for creating applications for the grants that were their primary funding source. There were years of grants on that computer that they often referred to or copied, using the same language or approach in the creation of new grants.
One night their computer was stolen, and along with it years of valuable intellectual property. I remember asking gingerly if she had a backup. Well, they certainly had intended to back up, but just never got around to it.
Lesson learned: Living without a backup is living dangerously. Sometimes you don’t realize the pain until the bad stuff happens.
A few years later, we were working with the accounting department of a smaller company. They were quite dutiful about backing up, actually. I find accounting folks are generally good about details like this. However, they kept the backups in a box right beside the computer so as not to forget. Again a theft occurred, and they lost their computer along with their box of backup disks.
Lesson learned: Keep your backups in a safe place. A backup is useless if stolen or destroyed. Keep backups in a secure place, free from harm, and consider keeping extra backups offsite. Don’t keep all your eggs in one basket.
One of our clients was incredibly dutiful about backing up. This was a large company who had every computer backed up every night over their network. Sounds good so far, right? Sometimes you seem to do everything right yet still have a poor outcome.
The hard drive on the server died. They called IT to ask for the most recent copy of their backup. “Absolutely,” IT responded, “it will take a couple of weeks to get it.” This system and its data were the lifeblood of the department, used constantly by a group of fifty people. Two weeks without this system meant two weeks of lost work.
As fastidious as IT was about backing up, they really had not thought about the importance of prompt file retrieval for their departments and users. The backups were stored offsite in a very secure environment, and it took time to get the proper storage medium and to retrieve the needed file.
Lesson learned: Make sure your backups are accessible, particularly those that are not under your direct control and those that you can’t live without.
We’ve all heard stories of backups gone bad. It happens. Backup tapes go bad, or the tape drives fail or go on the fritz, and the backups—although reported as successful—fail. Unfortunately, most folks discover these failures when they need the backups most, after a hardware failure of some sort. We’ve also seen examples of database files experiencing corruption but not exhibiting any symptoms of problems for weeks or months. When the administrator retrieves all the most recent backups, they are in fact corrupt as well. We’ve seen clients reverting to files as old as a month or more in order to find a good copy.
Lesson learned: Verify your backups from time to time. Restore your database file from a backup and test it. Make sure it works. Make sure the data is there. Personally, I think this is the most frustrating of situations. You do everything right and still get bit by a painful failure.
I’m writing this article because, yes, I recently heard yet another story of a lack of a backup. It was very painful for the person and company who contacted us, hoping we could magically retrieve their important business data from a failed hard drive. They likely have paper records from which they can recreate much of the info, but certainly not all.
Sometimes I’m asked how often a client should back up. My best answer is, “How much are you willing to lose?” It’s all about risk and reward. Think about how much pain your business will be in if you lost a week’s worth of data. What about if your database system disappeared completely? These are painful thoughts that require consideration.
Most of all, consider these questions now. Don’t wait for the bad stuff to happen. It’s too late then.