QA Hates You

Also in this month’s Software Test and Performance, Edward J. Correia talks about his experience with a Web site that’s suddenly national in scope, leading to a huge spike in visitors/users:

Watching the Today Show one morning in late January, I saw a segment about Rottenneighbor.com, a Web site relatively new at the time that lets people identify and rate their neighbors—good or bad— and post comments about them. The idea is to give people about to relocate a semblance of who lives nearby.

Those media blasts will spike your load far beyond what your normal operating parameters would indicate. The site normally runs 100 concurrent users and has tested out okay at 200 users will get to drink from the firehose after a piece on a morning program.

You’d better keep aware of whatever other marketing or PR is going on with your site to know when to worry.

I once worked on a site with rich media that was getting the full court press, including a Good Morning America segment. You’d better believe that your Director made sure that ship weathered the storm, including rejecting an ill-conceived “optimization” at the end of an all-nighter, werd.

(For a comment on another program that failed, see the post on the Simpsonize Me thing.)

Also, take note in Correia’s column about what application behavior he expects under load, and make it so.

You don’t want to piss off 100,000 bikers, do you?

The crash Saturday of the ticket-sales Web site for Harley-Davidson’s 105th Anniversary Celebration entertainment - Bruce Springsteen and the E Street Band - left many fans angry, empty-handed and wondering whether concert tickets were still available.

Tickets officially went on sale for the Aug. 30 concert at 8 a.m. Saturday, but the Web site crashed about 30 minutes later. Harley-Davidson reported Saturday evening that ticket sales will resume at 8 a.m. April 14 once the bugs are worked out.

Put that in your risk assessment plan.

As you well know by now, the Colorado Rockies server suffered a meltdown when World Series tickets came on sale. Rockies officials claim it was a malicious attack:

Colorado Rockies officials said Monday their computer system for online-only World Series ticket sales was the target of an “external malicious attack,” but online ticket sales were to resume at noon Tuesday.

Team spokesperson Jay Alves couldn’t immediately provide details of the attack.

You know what the attack was? Everyone wanted World Series tickets, and each customer was multiplied because as a group, they overcame limitations of 1 user 1 computer:

McLeod said he has Internet access from his apartment building but thought the library’s computers might be faster. His mother, father, uncle and girlfriend were trying to buy tickets from other computers, he said.

Or some overcame bandwidth exemptions:

Super-fast computers normally used only during emergencies were to be staffed Monday so state employees could buy Rockies World Series tickets online.

Until word of the plan got out, that is.

“I need volunteers to help push buttons in attempting access,” David Holm, recently the acting director of the state Division of Emergency Management, said in an e-mail obtained and released by KUSA-TV on Friday. “You will need to use break time, lunch time or leave time to do this and the only real perk I can offer right now is that if someone does not pay for their tickets within 3 days, you will get first crack at them.”

And here’s a guess: some ticket brokers or perhaps some developers came up with scripts that submitted orders to the Web site to improve their chances of getting tickets.

That’s not a “malicious attack.”

The servers failed from overwhelming demand. Of course, admitting that diminishes the brand, I suppose. But your company got to drink from the fire hose, Colorado Rockies; no one can fault you for failing to predict that your team would ever get to the World Series or what that could do to your systems.

Other stories on the breakdown:

• World Series ticket sales to resume after Colo. stall
• Surprise! Rockies’ online-only ticket sale plan implodes
• Spirit was willing, but servers were weak

And as a reminder, ungentle reader, your friend and mentor The Director would have found a problem here because he doesn’t run sissy load tests. However, no doubt the project would move forward even with the failure under extreme load because the stakeholders would be comfortable blaming DDOS.

There are many things one can learn from  this story:

Symantec Corp.’s early-warning system gave its enterprise customers a brief scare late Friday when it erroneously sent an alert that said an Internet-crippling attack was in progress.

However, the one I want to tease out is this lesson: always keep your junk data clean and readable and clearly defined as test data.

(more…)

Well, it’s not really Microsoft Warstrike, it’s Microsoft Web Application Stress Tool (WAST, which as you old Bard’s Tale fans know was the keystroke combination for the Warstrike conjuror’s spell, good for 4-16 points of damage on a group and a pretty potent weapon). Now, Microsoft Web Application Stress Tool lies buried in the Microsoft Downloads section (here). Screenshot:

Click for full size

It’s an old download (ca. 2002), but it looks to work with IE 7. So if you’re looking for a free rudimentary load-style tester, here’s something.

In this week’s SD Times, Larry O’Brien identifies a problem that load testing didn’t uncover:

Perhaps the only thing worse than a slow uptake of your application is a smash hit. Users have a way of outfoxing everything, including load tests, and the imperative to respond to existing customers can absorb all the working hours of a team that is scheduled to move on to the next version. Worse, when a product is exposed to an order of magnitude more users than planned and when the product is used more intensely than anticipated, the defect list grows rapidly, potentially panicking the team into treating the symptoms, not the causes. The resulting chaos can easily derail a team, especially one new to agile processes, where “the customer is always right” and being responsive were the values that led to the success in the first place.

Not long ago, I witnessed this very problem. I was engaged to work on the requirements and architecture of The Next Phase, which didn’t seem to have a lot to do with The Current Deployment, whose two big features were a comprehensive audit trail for management and a Web-based “dashboard” that gave users a much better view of their own context. Following the principles of “You Ain’t Gonna’ Need It” and “Don’t Repeat Yourself,” the dashboard and the auditing facilities used the same messages to request information; the dashboard, of course, stripped out the huge blobs of auditing data and presented a much-compressed summary. What was not anticipated (note the use of the passive tense to avoid blame) was that the users found the historical perspective of the dashboard very valuable and configured their dashboards to retrieve not just a day or two of history, but often everything they did in the past month.

Listen, there’s “load testing” and then there’s LOAD TESTING, and I would wager this organization did some “load testing.”

(more…)