I have seen many disaster recovery (DR) documents and prepared as many. In almost all of the DR recovery plans, I have seen that they did not have any single word about managing the non-technical side of the recovery, which are panic, pressure from top management and rumors that will surface in disaster time. Unfortunately there is no scientific way to test your staff’s nerves like testing your WAN connection. Here are some items to consider and include in your DR plan.
My long-time followers know that I am an oceangoing master mariner. The things I learned on board are perfectly applicable for the IT scenarios as well. The first and foremost thing you have to do is to stay calm. If you are in a management position, people will first look at you and try to get clues from you. It is of utmost importance to be calm. Even if you are uncomfortable, do not show it. Otherwise panic will reign and things will go out of control in an instant.
Then check if people are all right – both physically and emotionally. Make sure that all the people in your organization are all right. Is anybody missing? Does anybody need help? Count your staff and see them in person. Do not forget that an injured person is likely to say that he is in good shape. If you have any suspicion, ask for medical help.
Also make sure that everybody is emotionally in good condition. That means, make sure that every staff member is able to communicate with his/her family. Unless everybody contacts with their beloved ones – and preferably sees them – they will not be able to think about the company and the recovery plans. Forcing them will do more harm than good: the anxiety will not let them concentrate on the job at hand. This anxiety will result in making people more error prone, in a situation where error is one of the last things you would like to hear. To the best of your abilities, let them (and force them) to see their families, their beloved ones and come back with a clear mind.
In a disaster, you may lose a critical staff member. To be frank, as a manager you should have planned this in advance, and if you didn’t, you may find that this missing staff member is critical and does not have a second person who can step in. When making your disaster recovery plan, make sure that either every critical role has a second person (preferably all roles are “failover cluster”ed) or make sure that at least three consultants are listed to be called if that role goes missing.
Proper communication is an issue in itself. Almost all of the disaster recovery plans have a communication tree (or call tree if you will). The communication tree defines who reports to whom and where and who receives information from whom. In most of the plans, the reporting follows the company hierarchy: administrators report to their supervisors, the supervisors to managers and the managers to higher level executives. In the plan, the public communication role is also clearly defined – most probably the public relations department. The communication tree has to be reviewed by the staff quarterly. If not, people will revert to their communications habits and forget about the tree, especially in a disaster situation.
One of the things that cannot be omitted in the communication tree is to define how IT staff communicates. In all disaster situations, people are nervous and some of them will try to “get the latest.” Those people will try to find a way to ask their friends what is going on, then add their own opinions on top and spread rumors. There is no doubt that this misinformation will start harming the company both internally and externally. In many cases, the targeted IT staff member is a junior member: he will talk either because he is uncomfortable with someone higher than him in the hierarchy calling him asking for news or he is too inexperienced to assess the situation correctly and tells the news without knowing the consequences. It is on the senior members’ and his manager’s responsibility to encourage him about following the communication tree strictly no matter who calls him – someone may even be impersonating a C level executive. Make sure that your communication plans address those types of issues.
The disaster can be an IT-related disaster. If this is the case, establish a “direct link” to the CIO and make sure that the CEO is fed with the correct information. In a disaster situation, especially the CEOs, will be in direct contact with the shareholders, partners and other related parties. It is important to inform him/her constantly in all the steps. This will allow him to answer questions he is asked.
In a disaster scenario, employees will receive calls from everyone; from the media to the curious distant family members who haven’t called them for decades but figured out how much they wanted to talk with them right after the disaster is public. As a manager, make sure that how you will respond to the questions you are asked. I recommend to work this with your public relations department to be ready: in any case the reporters will find your personal mobile phone number and do anything to receive a small bit of information. It is a good idea to direct the landline phone calls to your call center so that you can better focus on the tasks at hand.
Keep notes during the recovery period and note everything that was not thought of and did not go according to plans. The tasks, processes, communications can be well thought of but the human related issues may not be at par. There may even be issues that look 100% correct but may have negative effects on people’s morale when executed. Review the human side of the issues and make sure that those are covered and then the plans are updated accordingly.
Finally make sure that your disaster recovery plan is reviewed at least once annually. I know many companies have this written in their quality documents but do not do. Threats change. Operations change. People change. Systems change. And when the disaster strikes the unreviewed plan does not match the reality of the business: everything is changed, the plan didn’t.
References
- Featured image: By Greg Henshall (This image is from the FEMA Photo Library.) [Public domain], via Wikimedia Commons
