Tag Archives: vSphere

DR Test Results

Recently we performed our annual Disaster Recovery test.  We have learned something very valuable every year and tried to adjust our recovery plans accordingly, with this year being no different.  Even with all the new technology, DR still seems to be a tricky undertaking.

The first year we tested our plan we found that we really, really, REALLY, don’t want to restore Active Directory onto unlike hardware.  The following year we had gotten a node onto our MPLS cloud which allowed us to have a replicated AD server at the DR site.  This greatly reduced the problem of restoring AD.  The year after that we tested our phone system portion of the DR plan and discovered that working with the telco in a DR situation will be challenging at the very least.  In the two years since that second test there have been some major changes that removed our ability to have replicated AD, so we were back to square one on that front.

This year we thought we would do a restore of our two year old VMWare environment.  We had decided to keep the scope to restoring only “Tier 0” service.  This included VMWare ESX , Symantec BackupExec, vSphere, and AD.  Time permitting we planned to restore as many servers as possible beyond the Tier 0 that were the bare minimum.

In the last year we had made the choice to purchase Symantec’s BackupExec Agent for VMware Virtual Infrastructure (AVVI).  This is a BackupExec agent that allows you to backup VMWare Guest OS files directly through the ESX server and/or SAN.  The idea is that we would have our virtualized servers backed up to tape at the VMWare file level and that this would allow us to restore directly back to ESX. Continue reading DR Test Results

Disaster Recovery / Business Continuity

A large part of our PCI and SAS70 compliance is to maintain, and test, a comprehensive and viable Disaster Recovery / Business Continuity plan.  As part of this we will be conducting our annual test of the Technology Availability Plan of our DR plan this coming Friday.  A co-worker and I will be flying to Scottsdale, Arizona where our contracted Disaster Recovery Vendor has it’s data-center that is stipulated for us.

For this test we will be testing VMWare and our ability to recover our vSphere environment.  We will have 3 servers in the test.  The first will be a Windows machine that we will use to install our backup environment and restore data from tape.  The other two machines will be ESX servers that we will setup and configure as our VM hosts.  We will then restore vCenter Server from tape as well as several other critical servers that we call “Tier 0”.

Tier 0 for our DR Plan consists of critical servers that are required to bring the rest of our environment back online in a disaster.  These include, Active Directory, Backup, and a few other infrastructure services that are needed before anything else can be restored.

We hope to have a successful test, and also hope to uncover roadblocks before they become issues in a real world scenario.

Registry Hive Recovered?!

We’re experiencing a weird pop-up message on our BackupExec 2010 server.  Every morning (after backups have run) we get a string of errors from Windows indicating that there was an error with the Registry Hive.  The error reads like this:

“Registry hive (file:) C:\WINDOWS\vmware-SYSTEM\vixmntapiXX was corrupted and it has been recovered.  Some data might have been lost.”

This error message is there every morning, and there will be anywhere from 10-20 of them that we have to click through. The only thing that changes is the XX is a number that increments.  As near as I can tell there is nothing wrong with the system and there are no symptoms of trouble other than the messages.

Going off the error message itself, and the fact that BE was running without this error until I turned on VMWare backups, I suspect that this is an error with the Agent for VMware Virtual Infrastructure (AVVI) otherwise know as the VMWare Agent.  I’ve tried some Googling and haven’t come up with much relating to this error specifically.  At this point its really just an annoyance as we have not see anything that would indicate an issue.  I’m just crossing my fingers that restores of data from AVVI will actually work!

We’re doing our Disaster Recovery Test this Friday so we’ll know pretty quick if these VMWare backups will work or not!  I guess we’ll find out.

BackupExec 2010 AVVI

I’ve been working with Symantec’s new VMWare agent for Backup Exec 2010 for the past couple of months.  We were excited to have differential backup of VMDK files through vSphere when they announced the new version of the software.  In practice however this is a bit more involved than we bargained for.

Continue reading BackupExec 2010 AVVI

vSphere Enterprise Plus

So today, after our little surprise with Virtual SMP or vSMP in vSphere Enterprise, we put in our PO to purchase vSphere Enterprise Plus edition.  Not exactly an inexpensive upgrade when you consider we have 14 processor licenses for ESX.

Then to add a little cream on top the Service and Support piece is a little odd.  VMWare likes to prorate the support you already have.  It seems they use some mystery multiplier that is less than 1 to determine how far the support you have left goes.  We just renewed our support in february, and had to buy another two months worth to get us back to not having to renew earlier next year.  I suppose this isn’t too surprising as the new edition of the product has a higher price tag and therefore the S&S will be higher.

I am indeed looking forward to the Host Profiles as well as the Virtual Distributed Switches.  This will really help us simplify our configuration and really make it easy to switch out ESX hardware when the time comes.  The vSMP will also be nice so that we can have up to the max of 8 processors on any one Guest OS.

All in all a good upgrade, it was just a surprise that we ended up having to go to it.