Kevin's Blog Photography, technology, and whatever else

5Jan/120

Unintentional load test

I've been a little out of touch with this blog in the last month or so. Ever since Thanksgiving things have been crazy, especially at work with the busy season.

Over the last year we have made some great efforts to dramatically increase our stability as well as availability by increasing redundancy to remove single points of failure. This was on many levels including the networking layer by introducing an HA firewall pair, and an HA load balancer pair. We also built out our server infrastructure by implementing 3 web servers for the load balancing, as well as clustering our database hardware and our application server hardware. All of this was intended to be able to easily handle the load of the retail busy season, between Thanksgiving and New Year's weekend. To be able to really know how much we could handle we wanted to load test the infrastructure top to bottom.

You might also like:

21Oct/110

Virtual Load Balancer Appliance

In our production environment that we host at Rackspace we have an F5 Big IP load balancer.  This is an excellent  product that has way more features than we can ever hope to need.

One problem with this setup is that in our development and staging environments we do not have load balancing and this has caused some issues when moving to production.  Some of the issues we've had stem around session persitence and what server the sessions are landing on.  These can be hard to troubleshoot, and if you aren't seeing them in the Dev or QA processes you are debugging while in production which is not good.

It became pretty evident that we needed to make our staging environment as much like production as we could, so I started to poke around for a virtual load balancer.  After a bit of searching I found several that seemed to fit the need, but many of them were not free.  With a bit more digging I found that the Zeus Traffic Manager product has a developer licence that is free to use for non production environments.  This suits our needs very well as this is just for staging testing.

I downloaded the VMWare template and had the box up in running in no time.  The initial web config is quick and easy, and the developer license was good for 1 year.  After which I assume/hope I can still get another free one.

The web based configuration is clean an easy to understand.  I have never administered a Load Balancer myself, and even so I was able to get all of our staging sites up and running with thier own pools, healthchecks, session persistence settings and everything we have in production.  Our QA team tells me that the speed is noticably better and it has already helped us uncover some issues that we have been fighting with our production servers.

All in this has been a great addition and the best price you can hope for.

Take a look and let me know your experience

You might also like:

2Nov/100

Disaster Recovery / Business Continuity

A large part of our PCI and SAS70 compliance is to maintain, and test, a comprehensive and viable Disaster Recovery / Business Continuity plan.  As part of this we will be conducting our annual test of the Technology Availability Plan of our DR plan this coming Friday.  A co-worker and I will be flying to Scottsdale, Arizona where our contracted Disaster Recovery Vendor has it's data-center that is stipulated for us.

For this test we will be testing VMWare and our ability to recover our vSphere environment.  We will have 3 servers in the test.  The first will be a Windows machine that we will use to install our backup environment and restore data from tape.  The other two machines will be ESX servers that we will setup and configure as our VM hosts.  We will then restore vCenter Server from tape as well as several other critical servers that we call "Tier 0".

Tier 0 for our DR Plan consists of critical servers that are required to bring the rest of our environment back online in a disaster.  These include, Active Directory, Backup, and a few other infrastructure services that are needed before anything else can be restored.

We hope to have a successful test, and also hope to uncover roadblocks before they become issues in a real world scenario.

You might also like:

13May/100

GNS3, my new best friend

I've just been in a Cisco training for the last few days.  The facility is very clean, nice, and the training was very well laid out and valuable.

As I suspect is often the case with these types of environments there were a few technical issues with our lab equipment.  These classrooms are not dedicated to any one class so keeping the gear in the class is impractical.  There are a lot of moving parts and keeping everything straight and running is surely no small task.  The actual hardware is somewhere else and connected to the classroom via the network.  In our class there were issues with the connection to our gear and it was slowing down our progress.

The instructor very thoughtfully had already prepared an environment that would better facilitate the labs.  In comes GNS3.  I had not used, or even heard of, this little gem and I am sorry for it.  GNS3 is an open source "Graphical Network Simulator" that allows you to model network devices virtually. 

You might also like: