Tag Archives: DRP

Disaster recovery planning

This article describes a plan and implementation process for disaster recovery planning. The secret to success in our experience is to involve the local response team from the outset of the project.

Copyright 2006 D.Lieberman. This work is licensed under the Creative Commons Attribution License

The disaster recovery plan is designed to assist companies in responding quickly and effectively to a disaster in a local office and restore business as quickly as possible. In our experience, participation in the planning and implementation process is more important than the process itself and helps ensure that the local response teams understand what they need to do and that resources they need will be available.

Keywords

  • DRP – disaster recovery plan
  • BIT  business impact timeline
  • ERT emergency response team
  • BIA  business impact assessment
  • Countermeasures  physical or procedural measures we take in order to mitigate a threat
  • PRT primary response time; how long it takes (or should take) to respond (not resolve)
  • RRP  recovery and restore plan; recovery from the disaster and restore to original state

DR planning is not about writing a procedure, getting people to sign up and then filing it away somewhere. In the BIT (business impact timeline) we see a continuum of actions before and after an incident. In the pre-incident phase, the teams are built, plans are written, and preparedness is maintained with training and audit. After an incident, the team responds, recovers, restores service and assesses effectiveness of the plan.

drp_1.gif

T=ZERO is the time an incident happens. Even though one hopes that disaster will never strike, refresher training should be conducted every 6 months because of employee turnover and system changes and self-audits conducted by the ERT every 3 months.

Building the DR plan

Build the ERT

Assign a 2-person team in each major office (for small offices with one or two people, then the employee will do it himself) to be the ERT. The people in the ERT need to have both technical and social skills to handle the job. Technical skills means being able to call an IT vendor and being able to help the vendor diagnose a major issue such as an unrecoverable hard disk crash on an office file and print server. Social skills means staying cool under pressure and following procedure in major events such as fire, flooding or terror attack.

In addition to an ERT in each office, one ERT will be designated as “response manager”. The response manager is a more senior person (with a backup person) that will command the local teams during crisis, maintain the DRP documentation and provide escalation.

The local response team becomes involved and committed to the DRP by planning their responses to incidents and documenting locations of resources they need in order to respond and restore service.

DR Planning Pre-incident activities

Kickoff call

The purpose of the call is to introduce the DRP process and set expectations for the local ERT. Two days before the call, the local team will receive a PowerPoint presentation describing DRP, the implementation process and the BIA worksheet. At the end of the call, the team will take a commitment to fill out the worksheet and prepare for a review session on the phone one week later.

Business Impact Assessment (BIA)

In the BIA, the team lists possible incidents that might happen and assesses the impact of a disaster on the business. For example there are no monsoons in Las Vegas but there might be an earthquake (Vegas is surrounded by tectonic faults and number 3 in the US for seismic activity) and an earthquake could put a customer service center in Vegas out of business for several days at least.

Recover and Restore

Recovery is about the ERT having detailed and accessible information about backups – data, server, people and alternative office space. Within 30 days after a disaster, full service should be restored by the ERT working with local vendors and the response manager.
It may also be useful using http://www.connected.com for backup of data on the distributed PC’s and notebooks.

DR Plan Review

The purpose of the call is to allow each team to present their worksheet and discuss appropriate responses with the global response manager. Two days before the call, the teams will send in their BIA worksheet. The day after the call the revised DRP will be posted.

Filling out the DRP worksheets

There are two worksheets the BIA worksheet (which turns into the primary response checklist) and the RRP (recover and restore plan) worksheet, which contains a detailed list of how to recover backup resources and restore service.

Filling out the BIA worksheet.

In the BIA worksheet, the team lists possible incidents and assesses the impact of a disaster on the business. In order to assess the impact of a disaster on the business we grade incidents using a tic-tac-toe matrix.

drp_2.gif

The team will mark the probability and impact rating for an incident going across a row of the matrix. A risk might have probability 2 and impact 5 making it a 7, while another risk might have probability 1 and impact 3 making it a 4. Countermeasures would be implemented for the 7 risk before being implemented for the 4 risk.

BIA worksheet step by step
  • Add, delete and modify incidents to fit your business
  • Grade business impact using the “tic-tac-toe” matrix for each incident.
  • Set a primary response time (how quickly the ERT should respond not resolve)
  • Establish escalation path  escalate to local service providers and response manager within a time that matches the business impact. Escalate to local vendor immediately and escalate to response manager according to following guidelines:
    • Risk > 6 within 15
    • Risk <= 6 and > =4 within 60
    • Risk < 4 within 2 hours.

drp_3.gif

Filling out the RRP worksheet.

In the RRP worksheet, the team documents in detail how to locate and restore backups and how to access servers (in the network and physically).

drp_4.gif

Maintaining the DR plan

DR exercises

Once every 6 months, the response manager will run an unannounced exercise, simulating an emergency. In a typical DR exercise the local ERT will be required to:

  • Respond to a single emergency (for example earthquake)
  • Verify contents of RRP check list
  • Physically locate backups

 

Self-Audit

After completion of the ER plan the local response team needs to perform periodic self-audits. A member of the local ERT will schedule an audit once every 3 months and notify the response manager by email regarding the date.

  • The audit should take about 1 hour and will check documentation and backup readiness
  • Documentation readiness
    • Make sure telephone numbers of critical suppliers posted at entrance to office. Make sure numbers are current by calling.
    • Read primary response sheet
    • Wallet-sized cards with emergency phone numbers and procedures, to be carried by all employees.
    • Onboard list who is in the office today and who is traveling or on vacation
  • Backup readiness
    • Local backup files/tapes
Tell your friends and colleagues about us. Thanks!
Share this
rug salesmen

Why your IT vendor doesn’t want you to do a risk analysis

Did you ever have a feeling that your IT integrator was treating you like a couple of guys selling you a Persian rug?  “Take it now – it’s so beautfiful, just perfect for your living room, a steal  for only $10,000 and it’s on sale” and when you ask if it will last, they tell you “Why do you want it to last? Enjoy, use it in good health, wear it out quickly and come back to the store so that we can sell you Persian Rug 2012”.

I had a meeting with a long-time client today – I’ve developed some systems for them in the FDA regulatory and clinical trial management space. We met for lunch to discuss a new project which involved an extension to an existing multi-center study.

The question of disaster recovery planning and offsite backup came up and  they asked me what I thought about backing up their clinical trial data together with their office file backups taken by their outsourcing IT provider.

I said this is a very bad idea because while their IT contractor specializes in providing Microsoft Windows/Office support for small businesses, they just don’t have the know-how or security expertise for HIPAA compliant data storage.

In general, small business IT integrators are  behind the curve on data security, compliance, disaster recovery and application software security. Their job is to keep Microsoft SBS running smoothly and install anti-virus software, not mitigate data security and HIPAA compliance attacks. The typical SMB integrator mindset is dominated by the Microsoft monoculture, and I would not expect them to be able to analyze data security threats correctly.

Whenever I go somewhere – I’m always looking at things with a security perspective – open doors, windows – things that could be easily lifted. Who might be a threat. Storing clinical data with a bunch of Microsoft Office files is just too big a risk to take. The CEO accepted my recommendation to encrypt data on a secure, hardened virtual server instance in the cloud and monitor potential exposure to new emerging threats as their application and project portfolio evolves.

After lunch and getting back into the office, I realized that Risk analysis is a threat to IT vendors.

Not every security countermeasure is effective or even relevant for your company. This is definitely a threat to an IT vendor salesperson who must make quota.

I am a big proponent of putting vendor suggestions aside and taking some time to perform a business threat analysis (shameless plug for our business threat analysis services,  download our free white paper and learn more about Business Threat Modeling and security management). In a business threat  analysis you ignore technology for a week or 2 and systematically collect assets, threats, vulnerabilities …and THEN examine the cost-effective security countermeasures.

Your vendor wants to sell you a fancy $20,000 application security/database firewall, but it may turn out that your top vulnerability is from 10 contract field service engineers who shlep your company’s source code on their notebook computers. You can mitigate the risk of a stolen notebook by installing a simple security countermeasure – Free open-source disk encryption software for Windows Vista/XP, Mac OS X, and Linux.

Information security vendors often promote their backup/data loss prevention/data retention/application security products using a compliance boogeyman.

The marketing communications often reaches levels of the absurd as we can see in the following example:

NetClarity (which is a NAC appliance) claims that it provides “IT Compliance Automation” and that it “Generates regulatory compliance gap analysis and differential compliance reports” and “self-assessment, auditing and policy builder tools for Visa/Mastercard PCI, GLBA (sic), HIPAA, CFR21-FDA-11,SOX-404, EO13231 government and international (ISO270001/17799) compliance.”

A network access control appliance is hardly an appropriate tool for compliance gap analysis but asserting that a NAC appliance or Web application firewall automates SOX 404 compliance is absurd.

Sarbanes-Oxley Section 404, requires management and the external auditor to report on the adequacy of the company’s internal control over financial reporting. This means that a company has to audit, document and test important financial reporting manual and automated controls. I remember the CEO of a client a few years ago insisting that he would not accept any financial reports from his accounting department unless they were automatic output from the General Ledger system – he would not accept Excel spreadsheets from his controller, since he knew that the data could be massaged and fudged. If there was a bug in the GL or missing / incorrect postings he wanted to fix the problem not cut and paste it.

Appropriate, timely and accurate financial reporting has absolutely nothing to do with network access control.


But the best part is the piece on the NetClarity Web site that claims that their product will help “Deter auditors from finding and writing up IT Security flaws on your network”.

And I suppose this really proves my point best of all.

Information security vendors like NetClarity do not have any economic incentive to really reduce data security and compliance breaches that would reduce  sales, making it better business for them  (not for their customers) to sell ineffective products.

This raises an interesting question about information security business models – but that’s a topic best left to another post.

 

Tell your friends and colleagues about us. Thanks!
Share this