Virtual Disaster Recovery

The team of data recovery engineers and support technicians at Security House have been particularly busy recently due to the floods, but this blip does not take away the increasing demand for our expertise. The real growth in demand in this last 10 years for data / disaster recovery has been doubling recently as more servers with SSD and more virtual / cloud based servers fail. This is not because traditional hard drives are necessarily more reliable, it’s more about an unjustified belief a new solution will cure an old problem.

Many Servers and computer systems can suffer a “dirty shutdown” when power is lost. With many moving to virtual environments we are seeing a rise in the number of recoveries from virtual server environments. Just because a VM can be hosted on a cloud does not mean it cannot fail. In fact from what we see in the lab the incidence of server failure and data corruption / loss is actually increasing despite cloud backup provision.

In reality a vitual machine and its data volume can become corrupted if the physical server, SAN or NAS develops too many bad blocks at the hardware / physical storage level. These bad sectors can increase with power fluctuations or outages or even vibration harmonics transmitted through large scale arrays.

The result is files within the VM become corrupt and file system corrections by ChKDSK or FSck cannot always fix the problem. We have seen a situation where a robust physical server array with 12 enterprise class drives in RAID 5 with numerous VMs was severely compromised by running ChKDSK within the Virtual Server. Some 2million files were lost and took nearly 5 weeks to recover.

The combination of RAID, bad blocks, VM, ChKDSK and quantity of files within one folder is as extreme a situation as it gets in disaster recovery and points to the need for data / drive integrity management procedures to be implemented by those who support such environments.

The cloud is a great place for data to live, but lightening can strike and with the recent heavy rainfall and floods even server rooms and data centers can lose power even if they are not completely flooded.