Information Technology Contingency Plan (Template)

<Vendor>
<Information System Name> Version 1.0
May 2, 2012

Proprietary and Confidential For Authorized Use Only

Information Technology Contingency Plan

Prepared by

Identification of Organization that Prepared this Document	

<insert logo>	Organization Name		
	Street Address		
	Suite/Room/Building		
	City, State Zip		

Prepared for

Identification of Cloud Service Provider	

<insert logo>	Organization Name		
	Street Address		
	Suite/Room/Building		
	City, State Zip		

Executive Summary

This document supports Information Technology (IT) Contingency Plan requirements for the Federal Risk and Authorization Management Program (FedRAMP) and contains the ITContingency Plan for the <Information System Name>. An IT Contingency Plan denotes interim measures to recover IT services following an unprecedented emergency or system disruption. Interim measures include the relocation ofITsystems and services to an alternate site or the recovery of IT functions using alternate equipment at the primary site.

Document Revision History

Date	Version	Description	Author	
5/2/2012	1.0	Document Published	FedRAMP Office	

				

Table of Contents

About this document	9
Who should use this document?	9
How this document is organized	9
Conventions used in this document	9
How to contact us	10
Contingency Plan Approvals	11
Introduction and purpose	12
Applicable Laws and Regulations	12
Applicable Standards and Guidance	12
Information System Name and Identifier	13
Scope	13
Assumptions	14
Concept of Operations	14
System Description	14
Three Phases	14
Data Backup Readiness Information	15
Site Readiness Information	17
Roles and Responsibilities	18
Contingency Planning Director (CPD)	18
Contingency Planning Coordinator (CPC)	18
Outage and Damage Assessment Lead (ODAL)	19
Hardware Recovery Team	19
Software Recovery Team	19
Telecommunications Team	20
Procurement and Logistics Coordinator (PLC)	20
Security Coordinator	20
Plan Distribution and Availability	21
Line of Succession/Alternates Roles	21
Activation and Notification	21
Activation Criteria and Procedure	22
Notification Instructions	22
Outage Assessment	22
Recovery	23
Sequence of Recovery Operations	23

Recovery Procedures	23
Recovery Escalation Notices/Awareness	23
Reconstitution	24
Data Validation Testing	24
Functional Validation Testing	24
Recovery Declaration	24
User Notification	24
Cleanup	25
Returning Backup Media	25
Backing Up Restored Systems	25
Event Documentation	25
Contingency Plan Testing	26
APPENDIX A KEY PERSONNEL AND TEAM MEMBERS CONTACT LIST	27
APPENDIX B VENDOR CONTACT LIST	28
APPENDIX C.1 ALTERNATE STORAGE SITE INFORMATION	29
APPENDIX C.2 ALTERNATE PROCESSING SITE INFORMATION	30
APPENDIX C.3 ALTERNATE TELECOMMUNICATIONS PROVISIONS	31
APPENDIX D ALTERNATE PROCESSING PROCEDURES	32
APPENDIX E SYSTEM VALIDATION TEST PLAN	33
APPENDIX F CONTINGENCY PLAN TEST REPORT	34
APPENDIX G DIAGRAMS	35
APPENDIX H HARDWARE AND SOFTWARE INVENTORY	36
APPENDIX I SYSTEM INTERCONNECTIONS	37
APPENDIX J TEST AND MAINTENANCE SCHEDULE	38
APPENDIX K ASSOCIATED PLANS AND PROCEDURES	39
APPENDIX L BUSINESS IMPACT ANALYSIS	40

List of Tables

Table 3-1. Information System Name and Title	13
Table 2-1. Backup Types	15
Table 2-3. Backup System Components	16
Table 2-4. Alternate Site Types	17
Table 2-5. Primary and Alternate Site Locations	17
Table 6-1. Personnel Authorized to Activate the ITCP	22
Table 5-2. Cleanup Roles and Responsibilities	25
Table 5-3. Event Documentation Reponsibility	26

List of Figures
No table of figures entries found.

ABOUT THIS DOCUMENT
This document has been developed to provide guidance on how to participate in and understand the FedRAMP program.
Who should use this document?
This document is intended to be used by Cloud Service Providers (CSPs), Third Party Assessor Organizations (3PAOs), government contractors working on FedRAMP projects, government employees working on FedRAMP projects, and any outside organizations that want to make use of the FedRAMP Contingency Planning process.
How this document is organized
This document is divided into tensections. Most sections include subsections.

Section 1 describes the introduction section which orients the reader to the type and location of information contained in the plan.

Section 2 describes concept of operations and provides additional details about the information system, the three phases of the contingency plan (Activation and Notification, Recovery, and Reconstitution), and a description of the information system contingency plan roles and responsibilities.

Section 3 describes the Activation and Notification Phase and defines initial actions taken once a system disruption or outage has been detected or appears to be imminent. This phase includes activities to notify recovery personnel, conduct an outage assessment, and activate the plan.

Section 4 describes the Recovery Phase activities and focuses on implementing recovery strategies to restore system capabilities, repair damage, and resume operational capabilities at the original or new alternate location.

Section 5 describes the Reconstitution Phase which is the third and final phase of ITCP implementation and defines the actions taken to test and validate system capability and functionality.

Section 6 describes how the ITCP Test Plan.
Conventions used in this document
This document uses the following typographical conventions:

Italic
Italics are used for email addresses, security control assignments parameters, and formal document names.

Italic blue in a box

Instruction: This is an instruction to the individual filling out of the template.

Italic blue text in a blue boxindicates instructions to the individual filling out the template.
Bold
Bold text indicates a parameter or an additional requirement.

Constant width
Constant width text is used for text that is representative of characters that would show up on a computer screen.

<Brackets>
Bold bluetext in brackets indicates text that should be replaced with user-defined values. Once the text has been replaced, the brackets should be removed.

Notes
Notes are found between parallel lines and include additional information that may be helpful to the users of this template.

Note:	This is a note.

Sans Serif
Sans Serif text is used for tables, table captions, figure captions, and table of contents.
How to contact us
If you have questions about FedRAMP or something in this document, please write to:

info@fedramp.gov

For more information about the FedRAMP project, please see the website at: http://www.fedramp.gov.

CONTINGENCY PLAN APPROVALS

x		
x	
<Name>	<Date>
<Title>	System Owner
<Cloud Service Provider>		<Name>	<Date>
<Title>System Owner
<Cloud Service Provider>	

x		
x	
<Name>	<Date>
<Title>	System Owner
<Cloud Service Provider>		<Name>	<Date>
FedRAMP	Authorizing Official	

INTRODUCTION AND PURPOSE
Information systems are vital to <Cloud Service Provider> mission/business functions; therefore, it is critical that services provided by <Information System Name> are able to operate effectively without excessive interruption. This Information Technology Contingency Plan (ITCP) establishes comprehensive procedures to recover <Information System Name> quickly and effectively following a service disruption.

One of the goals of an IT Contingency Plan is to establish procedures and mechanisms that obviate the need to resort to performing IT functions using manual methods. If manual methods are the only alternative; however, every effort should be made to continue IT functions and processes manually.

The nature of unprecedented disruptions can create confusion, and often predisposes an otherwise competent IT staff towards less efficient practices. In order to maintain a normal level of efficiency, it is important to decrease real-time process engineering by documenting notification and activation guidelines and procedures, recovery guidelines and procedures, and reconstitution guidelines and procedures prior to the occurrence of a disruption. During the notification/activation phase, appropriate personnel are apprised of current conditions and damage assessment begins. During the recovery phase, appropriate personnel take a course of action to recover the <Information System Name> components a site other than the one that experienced the disruption. In the final, reconstitution phase, actions are taken to restore IT system processing capabilities to normal operations.
Applicable Laws and Regulations

The following laws and regulations are applicable to contingency planning:

Federal Information Security Management Act (FISMA) of 2002 [Title III, PL 107-347] Management of Federal Information Resources [OMB Circular A-130]
Records Management by Federal Agencies [44 USC 31]
Applicable Standards and Guidance
The following standards and guidance are useful for understanding contingency planning: Computer Security Incident Handling Guide [NIST SP 800—61, Revision 1]
     Contingency Planning Guide for Federal Information Systems [NIST SP 800-34,
Revision 1]
   Guide for Developing the Risk Management Framework to Federal Information Systems: A Security Life Cycle Approach [NIST SP 800-37, Revision 1]
   Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities [NIST SP 800-84]
   Information Security Continuous Monitoring for Federal Information Systems and Organizations [NIST SP 800-137]

   Recommended Security Controls for Federal Information Systems [NIST SP 800-53, Revision 3]
Risk Management Guide for Information Technology Systems [NIST SP 800-30]
     Technical Guide to Information Security Testing and Assessment [NIST SP 800-115]
Information System Name and Identifier

This ITCP applies to the <Information System Name> which has a unique identifier as noted in Table 3-1.
Table 3-1.Information System Name and Title

UniqueIdentifier	
Information System Name	
Information System Abbreviation	

Scope

This ITCP has been developed for <Information System Name>which is classified as a moderate-impact system, in accordance with Federal Information Processing Standards (FIPS)
199. FIPS 199 provides guidelines on determining potential impact to organizational operations and assets, and individuals through a formula that examines three security objectives: confidentiality, integrity, and availability.The procedures in this ITCP have been developed for a moderate-impact system and are designed to recover the<Information System Name>within
<Recovery Time Objective (RTO) hours>. The replacement or purchase of new equipment, short-term disruptions lasting less than <RTO hours>, or loss of data at the primary facility or at the user-desktop levels is outside the scope of this plan.

Note:	Recovery Time Objective (RTO) defines the maximum amount of time
that a system resource can remain unavailable before there is an unacceptable impact on other system resources and supported mission/business processes.

Instruction: Edit the below list to name other plans and circumstances that are related but are outside the scope of this ITCP.

This ITCP does not apply to the following situations:
Overall recovery and continuity of mission/business operations. The Business
Continuity Plan (BCP) and Continuity of Operations Plan (COOP) address continuity of business operations.

Emergency evacuation of personnel. The Occupant Emergency Plan (OEP) addresses employee evacuation.

Instruction: A list of default assumptions are listed in the section that follows. The assumptions should be edited, revised, and added to so that they accurately characterize the information system described in this plan.

Assumptions

The following assumptions have been made about the <Information System Name>:

    The Uninterruptable Power Supply (UPS) will keep thesystem up and running for <total number of seconds/minutes>
    The generators will kick in after <total number of seconds/minutes> from time of a power failure
    Current backups of the application software and data are intact and available at theoffsite storage facility in <City, State>
    The backup storage capability is approved and has been accepted by the JAB
    The <Information System Name>is inoperable if it cannot be recovered within <RTO hours>
Key personnel have been identified and are trained annually in their roles Key personnel are available to activate the ITCP
<Cloud Service Provider>defines circumstances that can inhibit recovery and reconstitution to aknown state

CONCEPT OF OPERATIONS
This section provides details about the<Information System Name>, an overview of the three phases of the ITCP (Activation and Notification, Recovery, and Reconstitution), and a description of the roles and responsibilities of key personnel during contingency operations.
System Description

Instruction: Provide a general description of the system architecture and components. Include a network diagram that indicates interconnections with other systems.Ensure that this section is consistent with information found in the System Security Plan. Provide a network diagram and any other diagrams in Appendix G.

Three Phases

This plan has been developed to recover and reconstitute the <Information System Name>using a three-phased approach. The approach ensures that system recovery and reconstitution efforts are performed in a methodical sequence to maximize the effectiveness of

the recovery and reconstitution efforts and minimize system outage time due to errors and omissions.The three system recovery phases consist of activation and notification, recovery, and reconstitution.

Activation and Notification Phase.Activation of the ITCP occurs after a disruption, outage, or disaster that may reasonably extend beyond the RTO established for a system. The outage event may result in severe damage to the facility that houses the system, severe damage or loss of equipment, or other damage that typically results in long-term loss.

Once the ITCP is activated, the information system stakeholders are notified of a possible long- term outage, and a thorough outage assessment is performed for the information system.
Information from the outage assessment is analyzed and may be used to modify recovery procedures specific to the cause of the outage.

Recovery Phase.The Recovery phase details the activities and procedures for recovery of the affected system. Activities and procedures are written at a level such that an appropriately skilled technician can recover the system without intimate system knowledge. This phase includes notification and awareness escalation procedures for communication of recovery status to system stakeholders.

Reconstitution.The Reconstitution phase defines the actions taken to test and validate system capability and functionality at the original or new permanent location. This phase consists of two major activities: validating data and operational functionality followed by deactivation of the plan.

During validation, the system is tested and validated as operational prior to returning operation to its normal state. Validation procedures include functionality or regression testing, concurrent processing, and/or data validation. The system is declared recovered and operational by upon successful completion of validation testing.

Deactivation includes activities to notify users of system operational status. This phase also addresses recovery effort documentation, activity log finalization, incorporation of lessons learned into plan updates, and readying resources for any future events.
Data Backup Readiness Information

A common understanding of data backup definitions is necessary in order to ensure that data restoration is successful. <Cloud Service Provider> recognizes different types of backups which have different purposes and those definitions are found in Table 2-1.

Description

Backup Type

Table 2-1. Backup Types

Backup Type	Description	
Full Backup	A full backup is the starting point for all other types of backup and contains all the data in the folders and files that are selected to be backed up. Because full backup stores all files and folders, frequent full backups result in faster and simpler restore operations	
Differential Backup	Differential backup contains all files that have changed since the last FULL backup. The advantage of a differential backup is that it shortens restore time compared to a full backup or an incremental backup. However, if you perform the differential backup too many times, the size of the differential backup might grow to be larger than the baseline full backup.	
Incremental Backup	Incremental backup stores all files that have changed since the last FULL, DIFFERENTIAL OR INCREMENTAL backup. The advantage of an incremental backup is that it takes the least time to complete. However, during a restore operation, each incremental backup must be processed, which could result in a lengthy restore job.	
Mirror Backup	Mirror backup is identical to a full backup, with the exception that the files are not compressed in zip files and they cannot be protected with a password. A mirror backup is most frequently used to create an exact copy of the source data.	

The hardware and software components used to create the <Information System Name>
backups are noted in Table 2-3.
Table 2-3. Backup System Components

System/Component	Description	
Software Used		
Hardware Used		
Frequency		
Backup Type		
Retention Period		

Current backups of the <Information System Name> system software and data are intact and available at the offsite storage facility located at:

<Site Name>
<Street Address>
<City, State, Zip Code>

Personnel who are authorized to retrieve backups from the offsite storage location, and may authorize the delivery of backups, are noted in Appendix C.1.

<Cloud Service Provider>maintains both an online and offline (portable) set of backup copies of the following types of data on site at their primary location:
    user-level information
    system-level information
    information system documentation including security information.

Site Readiness Information

<Cloud Service Provider> recognizes different types of alternate sites which are defined in Table 2-4.
Table 2-4. Alternate Site Types

Type of Site	Description	
Cold Sites	Cold Sites are typically facilities with adequate space and infrastructure (electric power, telecommunications connections, and environmental controls) to support information system recovery activities.	
Warm Sites	Warm Sites are partially equipped office spaces that contain some or all of the system hardware, software, telecommunications, and power sources.	
Hot Sites	Hot Sites are facilities appropriately sized to support system requirements and configured with the necessary system hardware, supporting infrastructure, and support personnel.	
Mirrored Sites	Mirrored Sites are fully redundant facilities with automated real-time information mirroring. Mirrored sites are identical to the primary site in all technical respects.	

Alternate facilities have been established for the <Information System Name>as noted in Table 2-5. Detailed information about the alternate processing site, the alternate storage site, and alternate telecommunications can be found in Appendix F.

Table 2-5.Primary and Alternate Site Locations

Designation	Site Name	Site Type	Address	
Primary Site				
Alternate Site				
Alternate Site				

Roles and Responsibilities

skills and knowledge.

<Cloud Service Provider> establishes multiple roles and responsibilities forresponding to outages, disruptions, and disasters for the <Information System Name>. Individuals who are assigned roles for recovery operations collectively make up the Contingency Plan Team and are are trained annually in their duties. Contingency Plan Team members are chosen based on their

Instruction: Describe each team and role responsible for executing or supporting system recovery and reconstitution. Include responsibilities for each team/role including leadership roles. FedRAMP has established default roles and a small set of default responsibilities which should be edited and modified to match the actual organizational role names, responsibilities, and associated duties for your organization.

The Contingency Plan Team consists of personnel who have been selected to perform the roles and responsibilities described in the sections that follow. All team leads are considered key personnel.
Contingency Planning Director (CPD)

The Contingency Planning Director (CPD) is a member of senior management and owns the responsibility for all facets of contingency and disaster recovery planning and execution.

The CPD performs the following duties:

Makes the decision on whether or not to activate the ITCP Provides the initial notification to activate the ITCP Reviews and approves the ITCP
Reviews and approves the Business Impact Analysis (BIA)
Notifies the Contingency Plan Team leads and members as necessary Advises other Contingency Plan Team leads and members as appropriate
Issues a recovery declaration statement after the system has returned to normal operations Designated as key personnel
Contingency Planning Coordinator (CPC)

The CPC performs the following duties:

Develops and documents the ITCP under direction of the CPD Uses the BIA to prioritize recovery of components
Updates the ITCP annually
Ensures that annual ITCP training is conducted Facilitates periodic ITCP testing exercises Distributes copies of the plan to team members
Authorizes travel and housing arrangements for team members Manages and monitors the overall recovery process
Leads the contingency response effort once the plan has been activated

Advisesthe Procurement and Logistics Coordinator on whether to order new equipment Receives updates and status reports from team members
Sends out communications about the recovery Advises the CPD on status as necessary Designated as key personnel
Outage and Damage Assessment Lead (ODAL)

The ODAL performs the following duties:

Determines if there has been loss of life or injuries
Assesses the extent of damage to the facilities and the information systems Estimates the time to recover operations
Determines accessibility to facility, building, offices, and work areas Assesses the need for and adequacy of physical security/guards
Advises the Security Coordinator that physical security/guards are required Identifies salvageable hardware
Maintains a log/record of all salvageable equipment Supports the cleanup of the data center following an incident Develops and maintains a Damage Assessment Plan Estimates levels of outside assistance required
Reports updates, status, and recommendations to the CPC Designated as key personnel
Hardware Recovery Team

The hardware recovery team performs the following duties:

Installs hardware and connects power Runs cables and wiring as necessary
Makes arrangements to move salvageable hardware to other locations as necessary Ensures electrical panels are operational
Ensures that fire suppression system is operational Communicates with hardware vendors as needed (Appendix B) Creates log of missing and required hardware
Advises the PLC if new hardware should be purchased Connects network cables
Connects wireless access points
Software Recovery Team

The software recovery team performs the following duties:

Installs software on all systems at alternate site
Performs live migrations to alternate site prior to predictable disasters and outages

Installs servers in the order described in the BIA (Appendix L) Communicate with software vendors as needed (Appendix B) Advises the PLC if new software needs to be purchased Creates log of software installation problems
Restore systems from most current backup media
Maintains current system software configuration information in an off-site storage facility
Telecommunications Team

The Telecomm team performs the following duties:

Assesses the need for alternative communications Communicates Internet connectivity requirements with providers Communicates with telephone vendors as needed
Establishes communications between the alternate site and the users
Coordinates transportation of salvageable telecomm equipment to the alternative site Plans for procuring new hardware and telecommunication equipment
Advises the PLC if new equipment needs to be purchased
Retrieving communications configuration from the off-site storage facility
Plans, coordinates and installs communication equipment as needed at the alternate site Maintains plan for installing and configuring VOIP
Maintains current telecommunications configuration information at off-site storage facility.
Procurement and Logistics Coordinator (PLC)

The PLCperforms the following duties:

Procures new equipment and supplies as necessary
Prepares, coordinates, and obtains approval for all procurement requests Authorizes purchases up to <$ amount> for recovery operations Ensures that equipment and supplies are delivered to locations Coordinates deliveries
Updates the CPC with status
Workswith the CPC to provide transportation for staff as needed
Ensuring details of administering emergency funds expenditures are known. Processes requests for payment of all invoices related to the incident Arranging for travel and lodging of staff to the alternate site as needed Procures telephone equipment and leased lines as needed
Procures alternate communications for teams as needed.
Security Coordinator

The Security Coordinator performs the following duties:

Provides training for employees in facility emergency procedures and measures Providesphysical security, access control, and safety measures to support recovery effort Cordons off the facility including offices to restrict unauthorized access
Coordinates with the building management and theCPCfor authorized personnel access Coordinates and manages additional physical security/guards as needed
Acts as a liaison with emergency personnel, such as fire and police departments Provides assistance to officials in investigating the damaged facility/site
Ensures that data room/center at alternate site has locks (access controls) on the doors Coordinates and secures the transportation of files, reports, and equipment in coordination with theCPC.
PlanDistribution and Availability

During a disaster situation, the availability of the contingency plan is essential to the success of the restoration efforts. The Contingency Plan Team has immediate access to the plan upon notification of an emergency. The Contingency Plan Coordinator ensures that a copy of the most current version of the Contingency Plan is maintained at the <Cloud Service
Provider’s>facility.This plan has been distributed to all personnel listed in Appendix A.

Contingency Plan Team members are obligated to inform the Contingency Planning Coordinator, if and when, they no longer require a copy of the plan. In addition, each recipient of the plan is obligated to return or destroy any portion of the plan that is no longer needed and upon termination from <Cloud Service Provider>.
Line of Succession/Alternates Roles

The <Cloud Service Provider>sets forth an order of succession, in coordination with the order set forth by the organization to ensure that decision-making authority for the <Information System Name> ITCP is uninterrupted.

In order to preserve the continuity of operations, individuals designated as key personnel have been assigned an individual who can assume the key personnel’s position if the key personnel is not able to perform their duties. Alternate key personnel are named in a line of succession and are notified and trained to assume their alternate role, should the need arise. The line of succession for key personnelcan be found in Appendix A.

ACTIVATION AND NOTIFICATION
The activation and notification phase defines initial actions taken once a <Information System Name>disruption has been detected or appears to be imminent. The Recovery Time Objective (RTO) defines the maximum amount of time that a system resource can remain unavailable before there is an unacceptable impact on other system resources, supported mission/business processes, and the maximum tolerable downtime. Determining the information system resource RTO is important for selecting appropriate technologies that are best suited for meeting the

maximum tolerable downtime.This phase includes activities to notify recovery personnel, conduct an outage assessment, and activate the ITCP.

At the completion of the Activation and Notification Phase, key <Information System Name>ITCP staff will be prepared to perform recovery measures to restore system functions.

Activation Criteria and Procedure

The <Information System Name>ITCP may be activated if one or more of the following criteria are met:

The type of outage indicates <Information System Name>will be down for more than
RTO hours>
The facility housing <Information System Name>is damaged and may not be available within <RTO hours>
Other criteria, as appropriate.

Personnel/roles listed in Table 6-1 are authorized to activate the ITCP.
Table 6-1.Personnel Authorized to Activate the ITCP

Name	Title and ITCP Role	Contact Information	

			

NotificationInstructions

Instruction: Describe established notifications procedures. Notification procedures should include who makes the initial notifications, the sequence in which personnel are notified and the method of notification (e.g., email blast, call tree, text messaging, automated notification system, etc.).

Contact information for key personnel is located in Appendix A.
Outage Assessment

Company Sensitive and Proprietary Page 22

Following notification, a thorough outage assessment is necessary to determine the extent of the disruption, any damage, and expected recovery time. This outage assessment is conducted by<role name>. Assessment results are provided to the Contingency Planning Coordinator to assist in the coordination of the recovery effort.

Instruction: Outline detailed procedures to include how to determine the cause of the outage; identification of potential for additional disruption or damage; assessment of affected physical area(s); and determination of the physical infrastructure status, IS equipment functionality, and inventory. Procedures should include notation of items that will be needed to be replaced and estimated time to restore service to normal operations.

RECOVERY
The recovery phase provides formal recovery operations that begin after the ITCP has been activated, outage assessments have been completed (if possible), personnel have been notified, and appropriate teams have been mobilized. Recovery phase activities focus on implementing recovery strategies to restore system capabilities, repair damage, and resume operational capabilities at the original or an alternate location. At the completion of the recovery phase,
<Information System Name>will be functional and capable of performing the functions identified in Section 4.1 of the plan.
Sequence of Recovery Operations
The following activities occur during recovery of <Information System Name>:

Instruction:Modify the following list as appropriate for the system recovery strategy.

Identify recovery location (if not at original location)
Identify required resources to perform recovery procedures
Retrieve backup and system installation media
Recover hardware and operating system (if required)
Recover system from backup and system installation media
Implement transaction recovery for systems that are transaction-based.
Recovery Procedures
The following procedures are provided for recovery of <Information System Name>at the original or established alternate location. Recovery procedures are outlined per team and should be executed in the sequence presented to maintain an efficient recovery effort.

Instruction: Provide general procedures for the recovery of the system from backup media. Specific keystroke-level procedures may be provided in an appendix. If specific procedures are provided in an appendix, a reference to that appendix should be included in this section. Teams or persons responsible for each procedure should be identified.

Recovery Escalation Notices/Awareness
Notifications during recovery include problem escalation to leadership and status awareness to system owners and users. This section describes the procedures for handling escalation notices which defines and describes the events, thresholds, or other types of triggers that may be necessary for additional action.

Instruction: Provide appropriate procedures for escalation notices during the recovery efforts. Teams or persons responsible for each escalation/awareness procedure should be identified.

Company Sensitive and Proprietary Page 23

RECONSTITUTION
Reconstitution is the process by which recovery activities are completed and normal system operations are resumed. If the original facility is unrecoverable, the activities in this phase can also be applied to preparing a new permanent location to support system processing requirements. A determination must be made on whether the system has undergone significant change and will require reassessment and reauthorization. The phase consists of two major activities: validating successful reconstitution and deactivation of the plan.

Concurrent processing is the process of running a system at two separate locations concurrently until there is a level of assurance that the recovered system is operating correctly and securely.
Data Validation Testing

Instruction: Describe procedures for testing and validation of data to ensure that data is correct and up to date as of the last available backup. Teams or persons responsible for each procedure should be identified.An example of a validation data test for a moderate-impact system would be to compare a database audit log to the recovered database to make sure all transactions were properly updated. Detailed data test procedures may be provided in Appendix E, System Validation Test Plan.

Validation data testing is the process of testing and validating data to ensure that data files or databases have been recovered completely at the permanent location.

Functional Validation Testing

Instruction: Describe procedures for testing and validation functional and operational aspects of the system.

Functionality testing is a process for verifying that all system functionality has been tested, and the system is ready to return to normal operations.
Recovery Declaration
Upon successfully completing testing and validation, the <role name> will formally declare recovery efforts complete, and that <Information System Name> is in normal operations.
<Information System Name> business and technical POCs will be notified of the declaration by the Contingency Plan Coordinator.The recovery declaration statement notifies the Contingency Plan Team and executive management that the <Information System Name>has returned to normal operations.
User Notification

Instruction: Describe the notification procedures. Ensure that the procedures described are consistent with Service Level Agreements and contracts.

Company Sensitive and Proprietary Page 24

After the recovery declaration statement is made, notifications are sent to users and customers. Notifications to customers are made in accordance with predetermined notification procedures.

Cleanup

Instruction: Describe cleanup procedures and tasks including cleanup roles and responsibilities. Insert cleanup responsibilities in Table 5-1. Add additional rows as needed.

Cleanup is the process of cleaning up or dismantling any temporary recovery locations, restocking supplies used, returning manuals or other documentation to their original locations, and readying the system for a possible future contingency event.
Table 5-2. Cleanup Roles and Responsibilities

Role	Cleanup Responsibilities	

Returning Backup Media
It is important that all backup and installation media used during recovery be returned to the offsite data storage location. The following procedures should be followed to return backup and installation media to its offsite data storage location.

Instruction: Provide procedures for returning retrieved backup or installation media to its offsite data storage location. This may include proper logging and packaging of backup and installation media, preparing for transportation, and validating that media is securely stored at the offsite location.

Backing Up Restored Systems
As soon as reasonable following recovery, the system should be fully backed up and a new copy of the current operational system stored for future recovery efforts. This full backup is then kept with other system backups. The procedures for conducting a full system backup are:

Instruction:Provide appropriate procedures for ensuring that a full system backup is conducted
within a reasonable time frame, ideally at the next scheduled backup period.

Event Documentation
It is important that all recovery events be well-documented, including actions taken and problems encountered during the recovery and reconstitution effort. Information on lessons learned should be included in the annual update to theITCP. It is the responsibility of each ITCP team or person to document their actions during the recovery event.

Instruction: Provide details about the types of information each ITCP team member is required to provide for the purpose of updating the ITCP. Types of documentation that should be generated and collected after a recovery operation include: activity logs (including recovery steps performed and by whom, the time the steps were initiated and completed, and any problems or concerns encountered while executing activities); functionality and data testing results; lessons learned documentation; and an After Action Report.

Table 5-3.Event DocumentationResponsibility

Role Name	Documentation Responsibility	
	Activity log	
	Functionality and data testing results	
	Lessons learned	
	After Action Report	

CONTINGENCY PLAN TESTING
Contingency Plan operational tests of the <Information System Name> are performed annually. A Contingency Plan Test Report is documented after each annual test. A template for the Contingency Plan Test Report is found in Appendix F.

Note:	Please refer to NIST SP 800-34, Revision 1, Section 5, for guidance on Contingency Plan Testing.

Instruction: Please describe the procedures for the annual contingency plan testing. Include a description of the required test environment. Operational tests typically include the following:
Restore files from backup tapes
Verify that backup tapes are stored at designated off-site locations Determine whether data stored on backup tapes is valid and retrievable Perform failover testing
Test the UPS to ensure that it operates correctly in the event of a power disruption; Test the offsite backup vendor’s delivery response timeliness of media during normal daytime hours and during nighttime hours
    Test to ensure that offsite storage vendor only supplies backup tapes to authorized individuals
Test the generators to ensure that they turn on automatically
Perform call tree exercises to ensure that employees can be reached in a timely manner.

Whatever methods you use to test your plan, please describe those tests in this section.

APPENDIX A KEY PERSONNEL AND TEAM MEMBERS CONTACT LIST

Instruction: All key personnel (and their alternates) and Contingency Plan Team members should be noted on this contact list. The ITCP should be distributed to everyone on this list.

Role	Name and Home Address	Email	Phone	
Contingency Plan Director			Primary: Alternate:	
Alternate Contingency Plan Director			Primary: Alternate:	
Contingency Plan Coordinator			Primary: Alternate:	
Alternate Contingency Plan Coordinator			Primary: Alternate:	
Outage and Damage Assessment Lead			Primary: Alternate:	
Alternate Outage and Damage Assessment Lead			Primary: Alternate:	
Procurement and Logistics Coordinator			Primary: Alternate:	
Alternate Procurement and Logistics Coordinator			Primary: Alternate:	

APPENDIX B VENDOR CONTACT LIST

Vendor	Product or ServiceLicense #, Contract #, Account #, or SLA	
Phone	
		Primary: Alternate:	
		Primary: Alternate:	
		Primary: Alternate:	
		Primary: Alternate:	
		Primary: Alternate:	
		Primary: Alternate:	

APPENDIX C.1 ALTERNATE STORAGE SITE INFORMATION

Address of alternate storage site		
Distance from primary facility		
Is alternate storage facility owned by the organization or is a third-party storage provider?		
Points of contact at alternate storage location		
Delivery schedule and procedures for packaging media for delivery to alternate storage facility		
Procedures for retrieving media from the alternate storage facility		
Names and contact information for those persons authorized to retrieve media		
Potential accessibility problems to the alternate storage site in the event of a widespread disruption or disaster (e.g. roads that might be closed, anticipate traffic)		
Mitigation steps to access alternate storage site in the event of a widespread disruption or disaster		
Types of data located at alternate storage site, including databases, application software, operating systems, and other critical information system software		

APPENDIX C.2 ALTERNATE PROCESSING SITE INFORMATION

Alternate Processing Site:	
Address		
Distance from primary facility		
Alternate processing site is owned by the organization or is a third-party site provider		
Point of Contact		
Procedures for accessing and using the alternate processing site, and access security features of alternate processing site		
Names and contact information for those persons authorized to go to alternate processing site		
Type of Site (from Table 2-4)		
Mitigation steps to access alternate processing site in the event of a widespread disruption or disaster		

APPENDIX C.3 ALTERNATE TELECOMMUNICATIONS PROVISIONS

Alternate Telecommunications	
Name and contact information of alternate telecommunications vendors by priority		
Agreements currently in place with alternate communications vendors		
Contracted capacity of alternate telecommunications		
Names and contact information of individuals authorized to implement or use alternate telecommunications		

APPENDIX DALTERNATE PROCESSING PROCEDURES

Instruction: This section should identify any alternate manual or technical processing procedures available that allow the business unit to continue some processing of information that would normally be done by the affected system. Examples of alternate processes include manual forms processing, input into workstations to store data until it can be uploaded and processed, or queuing of data input.

APPENDIX ESYSTEM VALIDATION TEST PLAN

Instruction: Describe the system acceptance procedures that are performed after the system has been recovered and prior to putting the system into full operation and returned to users. The System Validation Test Plan may include the regression or functionality testing conducted prior to implementation of a system upgrade or change. Edit (or replace) the sample validation test plan provided to reflect the actual validation test plan for the system.

Procedure	Expected Results	Actual Results	Successful?	Performed by	
At the Command Prompt, type in sysname	System Log-in Screen appears				
Log-in as user testuser, using password testpass	Initial Screen with Main Menu shows				
From menu, select
5-Generate Report	Report Generation Screen shows				
Select Current Date	Report is generated on				
Report	screen with last successful				
Select Weekly	transaction included				
Select To Screen					
Select Close	Report Generation Screen Shows				
Select Return to Main Menu	Initial Screen with Main Menu shows				
Select Log-Off	Log-in Screen appears				

APPENDIX F CONTINGENCY PLAN TEST REPORT

Instruction: This section should include a summary of the last Contingency Plan Test. The actual procedures used to test the plan should be described in Section 6, not here.

Test Information	Description	
Name of Test		
System Name		
Date of Test		
Team Test Lead and Point of Contact		
Location Where Conducted		
Participants		
Components		
Assumptions		
Objectives	Assess effectiveness of system recovery at alternate site Assess effectiveness of coordination among recovery teams Assess systems functionality using alternate equipment Assess performance of alternate equipment
Assess effectiveness of procedures
Assess effectiveness of notification procedures	
Methodology		
Activities and Results (Action, Expected Results, Actual Results)		
Post Test Action Items		
Lessons Learned and Analysis of Test		
Recommended Changes to Contingency Plan Based on Test Outcomes		

APPENDIX GDIAGRAMS

Instruction: Insert network diagrams, data flow diagrams, and any relevant component diagrams here. All of the diagrams used should be consistent with those found in the System Security Plan.

APPENDIX H HARDWARE AND SOFTWARE INVENTORY

Instruction: Insert a hardware and software inventory here. The inventory should be consistent with the one found in the System Security Plan.

APPENDIX ISYSTEM INTERCONNECTIONS

Instruction: Provide a systemInterconnection Table which should be consistent with the Interconnections Table found in the System Security Plan. The Interconnections Table from the System Security Plan can be copied and pasted into this Appendix.

APPENDIX J TEST AND MAINTENANCE SCHEDULE

Instruction: All ITCPs should be reviewed and tested at least annually or whenever there is a significant change to the system. Provide information and a schedule for the testing of the system. For moderate-impact systems, a yearly functional test is required.

Instruction: ITCPs for other systems that either interconnect or support the system should be identified in this Appendix. The most current version of the ITCP, location of ITCP, and primary point of contact (such as the ITCP Coordinator) should be noted.

APPENDIX K ASSOCIATED PLANS AND PROCEDURES

System Name	Plan Name	

		

Instruction: Insert your Business Impact Analysis here. Please see NIST SP 800-34, Revision 1 for more information on how to conduct a Business Impact Analysis.

APPENDIX L BUSINESS IMPACT ANALYSIS