At this point we were getting frustrated and decided to try a file redirection restore to the local W2k3 server. The AVVI restore feature also allow for this, so we selected a local volume to restore our VM data to with the plan to then move those files over to ESX manually in the VIC client. Symantec then started throwing some other error and was failing those jobs as well. We then decided it would be best to initiate a support call with Symantec. We had the ticket be the highest priority we could without it being production down.
While we were on hold with Symantec we decided to switch to Plan B. This strategy was to manually restore VCB (VMWare Consolidated Backup) backups of selected servers via a hand carried USB hard drive, and to bring those into ESX by hand. In my official writeup, here is where I go into my acknowledgment that this is not a real DR test…
- It is unrealistic to assume these disk based backups would be readily available to restore from in a disaster given our current infrastructure. This however did provide valuable education as to the recovery process and obstacles to streamline our DR plan.
To farther complicate things, using VCB backups to restore is not an easy thing to do. There is a command line interface for restoring these backups in ESX, and the syntax is not very straightforward. It became quickly clear that we needed help, so we put in a call to VMWare support. (All the wile still on hold with Symantec for their support!) After a brief hold we were connected to a very knowledgeable tech. He tried walking us through everything we had already done and made some suggestions in the process. After about 30 minutes of trying, he suggested that we use VMWare Converter to import the VCB files. He assured us that this is his recommended method, and that he had high confidence that it would work. We installed the software, and kicked off the process.
While we were waiting for that to complete the Tech told us about a product that VMWare has that is designed with DR in mind. It is called VMWare Data Recovery (VDR). It is an appliance that you download from VMWare and run in your ESX environment. You configure it by giving it access to either a .vmdk file or a Windows share. From there you setup jobs to backup your VM Guests via a nice interface that installs in the VIC client. It has support for differential backups and also deduplicates your data in the storage. The best part is that having the Enterprise Plus license we are entitled to this software for free! We have downloaded the software and are planning on running some tests with it to determine if it would work better than our current method with AVVI.
After the discussion about VDR, and with about 60% complete on the converter import, we let the tech go as it was the end of his shift. At this point, after two and a half hours on hold and never speaking to a tech, we gave on Symantec Support.
To our relief the converter job did complete, and we were able to boot our imported VMs. Once the AD server was online in ESX we were able to add the backup server into the domain which we believe would make things much easier as we could then have an AD account as the restore login. We were also able to bring the vCenter Server online. And this is where we had run out of time. We had to catch a taxi to the airport to head home. If we were to have had more time we would have added the ESX servers into vCenter and attempted restoring more servers directly into ESX with AVVI.
In hindsight we are a bit disappointed with Syamtec’s AVVI at this point. We’ve previously noted that there are speed and reliability issues with the backups, but had the expectation that restores would go. Barring some sort of setting that we blatantly missed (possible!) there was clearly a technical issue with restoring to the ESX server when there was no vCenter server and no AD server. If you are in a disaster situation and don’t have any infrastructure running then you are really in trouble!
We are very hopeful that the VDR product from VMWare will help solve some of our issues. We are currently setting up a lab to replicate the DR Test so that we can evaluate if VDR will be the method moving forward. More about that setup in a post to follow!
I know this is an old post, but it’s uncanny how similar the network you’re talking about back in 2010 is to mine. Although probably much larger. I’m a one guy shop, 50 workstations, 15 servers. A little over half of which are running CentOS 6.7 and 7.0. The other half Windows 2016 AD. I’m backing up to an LTO5 library with BE2014. And my virtual infrastructure is ESXi 5.5 with a physical vSphere server on Windows 2008r2.
I’m very interested in your DR test. I’ve never run a DR test on my network, and I am VERY interested. As well as writing a DR plan for my company. Since your network was (back in 2010) I was hoping you might be able to offer myself some advice.
How did you solve the catalog problem? I remember back in BE 2010 or 2012, a catalog was an overnight process for myself. I had recently moved from LTO3 to 5 and had to catalog one of my LTO3 tapes in the library. Not fun.
As you might have noticed, much of my network/infrastructure is still circa 2011-2012. IT budgets at small companies don’t allow for a realistic upgrade schedule in all cases. What can a guy do? I have to work with the cards I’m dealt. With that said, you might guess I do not have the extra hardware laying around to run these tests. Obviously I can’t run the test on my production infrastructure. More virtual servers would push the current hardware past a comfortable level. Production servers would be effected. Any suggestions?
Any suggestions writing a good DR plan? Would it be best to write it as I’m running my first test? Any kind of help you can offer would be much appreciated.