Attack the Man Behind the Curtain

Ah. A Web application has stood up to your rigorous testing in the original test environment. Hey, in a strange quirk of fate, you’re living in that fantasyland where the application is then deployed to a staging server that resembles the structure of a production server so you can run through it again on a final build. The application passes with flying colors, which means it hasn’t bled to death in stage from your repeated thrusts. No, some project manager or customer lover comes in and says, “Ship it.” One of your technical co-workers deploys the application to production sometime in the middle of a night on a weekend or, more likely, 20 minutes before the client expects it.

You, tester, have just one run through it and MillerCoors time, right? Hold on, little pardner, not so fast. Particularly if your application’s production environment uses a load balancer.

A load balancer is an Internet thingy that sits between the Internet cloud and multiple Web servers set up and configured the same way. If your Web site or application expects much traffic at all, your company’s Gils and Neds will have set up a couple Web servers set up exactly the same way and put a load balancer in front of them. When traffic comes in from the Internet, all those sweet, sweet hitz your counters like hit that load balancer. To them, www.ourcompany.com is that load balancer. The load balancer picks from the Web servers that have the actual, you know, files that comprise the Web site on them and sends the individual visitor to the least busy among them. It balances the load, you know. Think of it like this:

The load balanced

Now, then. If you think you’re going to go to www.website.com and run through a quick smoke test and prove the site works, what you’ll prove is that the site works on one of the Web servers.

However, tech staff being what they are, the odds are too good (that is, the chance exists) that someone failed to copy dependencies to one of the Web servers. A JavaScript file here, a CSS file there, an image there, maybe even a whole PHP page that includes the new shopping cart.

You can test the individual Web servers either using the direct IP addresses of the Web server themselves (bearing in mind that domain name resolution will probably provide you with the IP address of the load balancer if you try cleverly to ping the Web site by its name to figure it out) or by configuring the hosts file on your test workstation to resolve to things like www1.mywebsite.com, www2.mywebsite.com, and www3.mywebsite.com. You should perform your smoke testing/sanity testing, automated link checking, and whatever sort of regression things you’d planned to do to the site on each of the servers.

If you don’t and those slops with the big salaries do bollix something up, your users will encounter the problem, but when you try to recreate it, you’ll have a 1 in n chance of encountering it if you follow concise steps provided by the user (assuming that the user provided steps to the help desk instead of a mere Cro-Magnon interspersal of obscenity and technical naivete).

Then, when you try to get a developer to fix it, he or she will try it on the local environment or the stage environment where it works, and then he or she will dismiss you dismissively and dis you disrespectfully and return to his or her Facebook page.

So be wise, kid, and plan on spending that extra time touching them all. A moment in the sun, John Fogerty would say if he were me instead of having a real job. When you’re testing a production environment using a load balancer, you’re not doing every thing, every time.

No Responses to “Attack the Man Behind the Curtain”

  1. gimlet Says:

    Good post. Just thought I’d add a few specifics for those developers whose heads aren’t spinning from the concept of the load balancer.

    Using a hosts file is an important testing tool, particularly if your webserver or app responds with the load-balanced URL (ie, http://www.mywebsite.com). I’ve seen too many developers go directly to a particular server, get redirected to the load-balanced URL, and not understand why things don’t work the way they expected them to. This can cause a flurry of angry, sophomoric Twitter updates.

    So, kids, here’s where your hosts file is:

    In Windows, the path to the hosts file is typically:

    C:\WINDOWS\system32\drivers\etc\hosts

    For Linux and other Unix systems:

    /etc/hosts

    For OS X:

    /private/etc/hosts


wordpress visitors