End User IT Disaster Recovery Lessons Learned in Katrina's Wake

Practical lessons from a user perspective

On August 29, 2005, Hurricane Katrina blew through New Orleans. With the hindsight of 18 months, here are some changes we implemented, and ideas we expanded based on our experience. We hope they help you develop your firm's disaster recovery/business continuity plans.

REVIEW AND UPDATE

It's critical to keep your DR/BC plan current and realistic. An outdated plan, or one that assumes the impossible, won't help you in real life. We now review our plan every six months, which helps us revise it to incorporate new technology or ideas. For example, if a new application is added to your network, someone should ensure that the data it produces is replicated, and that the application itself will be available remotely, and at your "hot site". Should this be overlooked at the time the new application is installed, the periodic review, if done properly, will catch the oversight and make the DR/BC plan accurate again.

DESIGNATE A REMOTE OFFICE

I can't say enough of the value in having a remote office ready to set up shop in after a disaster. Even if that office isn't large enough to accommodate the entire staff, it's a start and is better than nothing at all. After Katrina, our Baton Rouge branch office instantly became our main office.

Other firms that only had an office in New Orleans were still scrambling to find floor space while we were wrapping up our network restore. You can bet during a regional disaster like Katrina, available space will go very quickly. If you have multiple offices, designate one as the expected backup main office, and consider making it a hot site where you can continuously replicate your data. Be sure that the remote office is not so geographically close to your main office that it might be equally hit by calamity.

Alternatively, if your firm doesn't have a secondary office, consider a co-location arrangement with a vendor who has a data center within a few hundred miles of your main office, or close enough to drive to within a few hours. (Again, factor in possible regional events).

You might also consider a "mutual aid" partnership with another firm in a different region, obviously not a direct competitor, in the event of a major regional catastrophe. Don't forget to incorporate remote access tools into your hot site, especially if you think people will be dispersed and not all working at the hot site.

TEST, REVISE AND RE-TEST

Post-Katrina, we now schedule tests of our hot-site systems at least twice a year, tied to the beginning and end of hurricane season. Our goal is to ensure that we can duplicate network operations as closely as possible and according to our established plan.

It is important to realize that a hot site likely won't be an exact twin of your production network. It's important to have critical applications and data current and available; however, from our users' perspective there will be subtle, noncritical differences such as invalid printer names, and invalid shortcuts to programs that are not available. Document these differences well enough and incorporate back into your plan. This will ensure that when a real failover occurs, everyone involved will know what to expect.

Throughout the year, we also continuously perform data integrity tests of our replication software. While we have faith in the reliability of our replication system, and monitor its status, it's very easy to fire up the hot-site databases and perform a quick query to just be sure everything is in sync.

In fact, the software allows us to automate that process completely, all without ever stopping the real-time replication. You don't want to come to discover that your hot-site data is actually six months out of date.

PEOPLE AND COMMUNICATIONS

Do not forget about the most important component of your firm: the people.

The best technology won't be worth a dime without the human resources. After Katrina, many of our staff lost everything and were scattered throughout the region. Those who could make it to our Baton Rouge office did so as quickly as their situation allowed, while others were in other nearby cities.

In preparing your own disaster plan, most specific details regarding human resources can't be ironed out in advance. However, you can develop a protocol about what the firm will expect of employees and what employees can expect of the firm following a disaster, for example, how will the firm contact employees, and vice versa? Keep employee data updated and keep a printed hard copy offsite. Be sure your management team always has a copy. Also remember to include the contact list in any type of disaster "hotbox" you create.

Given the annual threat of hurricane evacuations, we went one step further and asked that employees also provide contact information about where they might evacuate (e.g., friends and family.) We now hold biannual meetings to remind empoyees about emergency procedures.

One specific thing we have done to help maintain communications during and immediately following a disaster is to establish service with a Web-based SMS (text-messaging) service.

Post-Katrina, Gulf Coast residents quickly discovered that though cellular service was nearly nonexistent, text messaging generally worked flawlessly. We will use this to broadcast vital information to employee cell phones via a simple Web interface.

RISKS AND PLAN

Understand your firm's specific threats and plan for those scenarios first. Although our firm is vulnerable to other disasters, being located in New Orleans (and having gone through Katrina) our disaster planning revolves around hurricane scenarios.

I would imagine folks on the West Coast would likely plan for earthquakes and wild fires. But also consider disasters that may only briefly interrupt your operations, such as a defective sprinker system, as well those that can potentially keep you out of your office for extended time, or completely destroy your main site. Each day when you leave your office, stop and think about what you would do if the next morning you discover the main office was completely destroyed.

In addition, it is your job to get a new office running by the end of the day, or sooner. Do you have servers? Do you have tapes? Do you have a disaster plan?

James Zeller is network manager at Chaffe McCall, in New Orleans, and won an honorable mention in the 2006 Law Technology News Awards' IT Director of the Year competition.

END

If you would like to know more about how your organisation can get involved and benefit from working with the Continuity Forum, please email us HERE! or call on + 44 (0) 208 993 1599.