The first disaster recovery exercise is a moment of truth for any organization. It’s not just a technical test; it’s a verification of organizational maturity and the true resilience of systems. A well-conducted exercise, even if it reveals unexpected criticalities, is a success because it transforms potential disasters into learned lessons. In my role as a Senior IT Consultant, I’ve had the opportunity to orchestrate and closely observe these simulations in complex environments, such as an organization with 2,000 workstations and over 300 VMware VMs. The objective was clear: validate the Disaster Recovery (DR) and Business Continuity (BC) plan in the face of a total primary datacenter outage. Despite meticulous planning, reality exceeded all expectations, highlighting a series of problems that only a practical test could reveal.
Tested on: VMware vSphere 7.0 infrastructure · Veeam Backup & Replication 12 · August 2026
Prerequisites / Test Environment
For the business continuity exercise, the test environment was configured to replicate the production infrastructure as closely as possible. This included two geographically separated sites: an active primary site and a passive Disaster Recovery (DR) site. The primary site hosted a VMware vSphere cluster with 300+ VMs, SAN storage, and complex network services managed by Cisco and FortiGate. The DR site had similar hardware but with a minimal configuration, awaiting activation. Backups were managed via Veeam Backup & Replication, with off-site replicas. The business continuity plan included detailed procedures for critical service failover, from Oracle databases to enterprise applications and document management systems. The simulation involved the total unavailability of the primary site, forcing the activation of the DR site and the restoration of services within the defined Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets. The main challenge was coordinating various teams (system administrators, DBAs, application developers) and validating existing documentation.
1. The Exercise Plan: Theory vs. Reality
The exercise plan was carefully drafted, including failover scenarios for key services. However, its practical implementation revealed a series of discrepancies. The first phase involved simulating a total blackout of the primary datacenter, with the goal of activating critical services on the DR site. This process should have followed a well-defined sequence, but difficulties emerged from the very first steps. The documentation, while existing, was not always aligned with the current infrastructure configurations. Commands and file paths had changed, rendering procedures unusable. This highlighted the importance of a continuous review and update process for documentation, especially in dynamic environments. Read also: PSN Cloud Migration: The Reality Behind the 2026 Mandate
A striking example was the restoration of a Linux-based application server. The procedure indicated a network configuration command that was no longer valid due to an operating system update:
# Obsolete command in the plan
ip addr add 10.0.0.10/24 dev eth0
# Correct command for the updated OS
nmcli connection modify "System eth0" ipv4.addresses 10.0.0.10/24
nmcli connection up "System eth0"This small error caused a significant delay, as the team had to diagnose and resolve the issue in the field, instead of following a tested procedure.
2. Backup Gaps and Data Inconsistencies
One of the most critical aspects that emerged was the presence of gaps in backups. Despite a robust backup system like Veeam, some crucial datasets had not been included in protection plans, or their retention policies were insufficient. This particularly affected small but critical Oracle databases and network shares used for application configurations. The restoration attempt revealed that data was either unavailable or too old to ensure an acceptable RPO. Read also: Backup Tools: Restic vs BorgBackup on 2TB
For Oracle databases, manual intervention was required to recover data from alternative backups, significantly extending the RTO. Backup verification is fundamental, but it often only checks for their existence, not their integrity and completeness. A command to check the consistency of a Veeam backup, for example, is:
Get-VBRJob | ForEach-Object { Get-VBRBackup -Job $_ | ForEach-Object { Start-VBRRestoreSession -Backup $_ -RestorePoint $_.GetLastRestorePoint() -RunAsync } }This script, although not a true restore, can help identify consistency issues at the VM level. However, it does not replace a full restore and data validation.
3. Team Coordination and Undefined Roles
The simulation highlighted the need for more effective team coordination. In stressful situations, communication can become fragmented, and roles less clear. Different teams (networking, virtualization, database, applications) acted independently, causing duplication of efforts and, in some cases, conflicting interventions. The lack of a clearly designated “Incident Commander” and a unified communication channel slowed down the recovery process. Read also: ACN Incident: 24-Hour Notification Checklist
The need for joint training sessions for all involved teams emerged, to simulate crisis scenarios and clearly define everyone’s responsibilities. An internal and external communication plan is equally crucial, to inform stakeholders and manage expectations.
4. Hidden Dependencies and Unexpected Recovery Times
Many seemingly independent services had hidden dependencies that compromised the RTO. For example, a critical web application depended on a secondary authentication service that was not restored in time, blocking user access. These dependencies had not been mapped in the DR plan, leading to unexpected delays. The complexity of an enterprise environment requires detailed mapping of all interdependencies between services and applications. Using Configuration Management Database (CMDB) tools or Application Dependency Mapping solutions can help identify these relationships. Read also: Proxmox Replication: Slow WAN, Low RPO
The RTO calculation proved to be overly optimistic. Initial estimates did not account for VM boot times, database synchronization, or manual service verification. The true RTO extended well beyond expectations, highlighting the need to test the end-to-end process and realistically revise timings.
Common Errors and Troubleshooting
The most common errors encountered during the exercise were outdated documentation, incomplete backups, and lack of coordination. To address these issues, implementing a continuous feedback loop is essential. Every infrastructure change (software update, new VM, configuration change) must trigger a review of the DR documentation and, if necessary, a targeted test. Using automation tools like Ansible for server configuration can help maintain consistency and reduce manual errors. For example, an Ansible playbook to configure a network interface ensures that the procedure is always the same and documented in the code:
- name: Configure network interface
ansible.builtin.nmcli:
conn_name: "System eth0"
ifname: eth0
type: ethernet
ip4_address: 10.0.0.10/24
state: present
autoconnect: trueFor backups, it is essential not only to verify that jobs are completed but also to periodically perform partial and full restores, validating data integrity. This includes restoring databases and starting restored VMs to test their functionality.
FAQ — Frequently Asked Questions
How often should we perform business continuity exercises?
Full exercises should be performed at least once a year. However, partial tests or those focused on individual critical services or components can be performed more frequently, for example, quarterly, to validate specific procedures or infrastructure updates. The frequency depends on the complexity of the environment and the speed of changes.
How can we ensure that documentation is always up to date?
Integrate documentation review into the Change Management process. Whenever a significant change is implemented in the infrastructure or services, the related DR procedure should be reviewed and updated. Using version control tools for documentation, such as Git, can help track changes and facilitate collaborative reviews.
What is the role of an Incident Commander during an exercise?
The Incident Commander is the single point of contact for incident or exercise management. They are responsible for coordinating all teams, internal and external communication, and critical decisions. Their role is to ensure the plan is executed efficiently and resources are allocated correctly. This figure must have authority and leadership skills.
How can we identify all dependencies between services?
Identifying dependencies can be complex. Start with a workshop involving all technical teams and application stakeholders to map data flows and interconnections. Using Application Dependency Mapping (ADM) tools can partially automate this process, providing a dynamic visualization of dependencies in complex environments. Architecture documentation is a crucial starting point.
Is it sufficient to test only the most critical services?
No, it is not sufficient. Although critical services have priority, a comprehensive exercise should also include less critical services that might have hidden dependencies or affect the recovery of primary services. A phased approach, starting with the most critical services and then expanding, can be effective, but the ultimate goal is to test the entire recovery chain.
Conclusions with Operational Takeaways
The first business continuity exercise, challenging as it was, was a success because it revealed criticalities that would otherwise have only emerged during a real disaster. The operational takeaways are clear: resilience is not just a technological matter, but an organizational one. It is crucial to invest in continuous documentation review, joint team training, and thorough dependency mapping. Backup verification must go beyond simple completion confirmation, including periodic restore tests. Finally, clear role definition and the establishment of an Incident Commander are non-negotiable elements for effective incident management. Only through a cycle of continuous testing, learning, and improvement can true corporate resilience be built.
Sources
Updated: August 2026
