Archive for the ‘Load testing’ Category

Exploratory Testing During Load Testing

Tuesday, March 29th, 2016 by The Director

In Connecticut, some exploratory testing types found and exploited a software flaw in lottery terminals:

An investigator for the Connecticut Lottery determined that terminal operators could slow down their lottery machines by requesting a number of database reports or by entering several requests for lottery game tickets. While those reports were being processed, the operator could enter sales for 5 Card Cash tickets. Before the tickets would print, however, the operator could see on a screen if the tickets were instant winners. If tickets were not winners, the operator could cancel the sale before the tickets printed.

It’s a condition that only occurred while the system was under processing load.

Which is why, whenever I get to do some load testing, I also like to call up the application under test and run through some basic smoke tests with it. You can find different places where resources are not available or where the load times can lead to unintended consequences–like allowing the user to click a button that renders but is hidden when the page fully loads. Or to act on data that the user should not be able to act on, as the lottery terminal displays.

Of course, you can do something like this through some network-throttling tools, but that will only really handle client-side slowdowns and problems, not necessarily issues with the server and infrastructure.

Also, it’s a way to get one more user’s worth of load on the system, and given our load testing budget most of the time, that can be a 5% increase over the 20 virtual users we have licenses for.

As If Millions of Prescription Change Orders Suddenly Cried Out in Terror and Were Suddenly Silenced

Thursday, January 12th, 2012 by The Director

You might know, if you’re in the United States, have health insurance, and have an insurer that uses Express Scripts for its prescription benefits management, that as of January 1, 2012, the Walgreens pharmacy chain and Express Scripts contract ended. Which means that your insurance card does not work at Walgreens, and if you want our pharmaceuticals at reduced rates, you have to transfer your prescriptions to other pharmacies.

How many prescriptions are affected?


An Express Scripts spokesman say their customers previously filled 90 million prescriptions at Walgreens. Now they’re taking them elsewhere.

It’s my understanding that the Express Scripts processing system was down all day on January 11. Is it related? I don’t know.

But I do wonder whether Express Scripts load tested its system to handle 90,000,000 change orders in a matter of days while handling its normal processing for all other normal maintenance.

It could be a valuable lesson anyway: Even after you’ve load tested your application to the limits of your budget for virtual users or to a level where the stakeholders are comfortable with very gradual ramp up times, sometimes events out of IT’s control could lead to a catastrophic meltdown.

Everybody Use Lower Case From Now On

Thursday, December 22nd, 2011 by The Director

Story: Hold those caps: The average web page is now almost 1MB:

Mobile broadband caps might not be putting the hurt on most mobile subscribers yet, but data usage is going to keep creeping up. And that’s without people doing any more actual browsing.

The HTTP Archive charted the growth of the average web page and found that average web pages have grown from 726 KB a year ago to 965 KB now. The 33 percent jump is due in large part to more images and third-party scripts like ads and analytics. Javascript content, spurred on by the rise of HTML5, has grown over the last year by 44.7 percent, according to analysis by Royal Pingdom.

I remember when 1Mb hard drives cost $1000, you damn kids. Before you were born.

But it does clarify an interesting point: The time your Web page takes to load isn’t the only metric you should check when you’re performance testing your mobile applications or even your regular Web site if you expect mobile users will access it.

Although your leadership might say, That’s okay, we’re not designing for mobile users, if your bloated Web site is chewing up their allocated bandwidth, they’re going to find someone more streamlined.

(Link via Tweet retweeted by Fred Beringer.)

Unanticipated, Except For Those Who Anticipate Things

Tuesday, December 6th, 2011 by The Director

At a little after 7am this morning, I tweeted:

In about an hour, I’ll be participating in the crowd-sourced load testing of Good luck, everyone.

A couple of near-timeouts later and some really, really long load times, and I’m the proud owner of one share of Green Bay Packers stock. It’s fewer shares than I own of the startup where I used to work, but they’re worth about the same.

At any rate, two hours after I tweeted, the first news reports about the slow Web response time appeared:

On Twitter, fans were reporting that the it was difficult to access the website – – or the toll-free line: 855-8-GOPACK.

Murphy and other team officials said they anticipated there would be high demand. “Have patience,” Murphy told reporters at a news conference Tuesday morning. “Be patient.”

Who could have expected that this would have occurred? Anyone with dramatic Web launch experience.

UPDATE: More on that great tsunami here:

Sarah Johnson, 34, of Portage said it took her nearly 20 minutes to complete what should have been a 30-second process, but it was worth to wait.

The team received 1,600 orders in the first 11 minutes of the sale, said Packers president Mark Murphy, who had to reassure fans the Packers website was still working. Team spokesman Aaron Popkey said he did not have any sales data as of early Tuesday afternoon.

“It’s just a question of volume,” Murphy said. “Fans are excited about this opportunity. We just encourage fans to be patient.”

150 orders a minute. What, were they running it in development?

Lady Gaga, Accidental Load Tester

Wednesday, May 25th, 2011 by The Director

Lady Gaga performs “Load Tested This Way“: Inc.’s one-day, 99-cent promotion of Lady Gaga’s highly anticipated second studio album, “Born This Way,” is resulting in downloading delays on the Internet retailer’s website due to high volume, the company said Monday.

That, my friends, is why you should always load test to the maximum, not just to the comfortable. Because an event that brings down your site or service is not going to fit in the comfortable range.

Load Testing Asterisks

Friday, August 20th, 2010 by The Director

I’ve recently engaged to do some load testing oversight, and as such, this required some different asterisks for my estimate than my regular functional testing estimates. Some are applicable to load testing in general, but some are specific to the role I’ve taken as sort of a project manager wielding a prod over a third company that will provide the actual scripting and running of the tests.

Here are the asterisks I came up with, the torpedoes that could sink the project or, at the very least, the estimate:

  • Problems with the environment or application require additional work. If the testing environment is not configured correctly or does not mirror the anticipated production environment, the load tests might require additional starting/stopping and run times.
  • Functional defects impede test scripts. If the test scripts encounter functional defects, the load tests might require additional iterations after defects have been corrected.
  • Functional workflow remains the same between load tests. This estimate anticipates that corrections to any defects/load issues found will not require refactoring or rewriting the test scripts. If the application changes in such a fashion, the test team will need additional time to adjust and retest the scripts.
  • Communication between test liaison and test vendor runs smoothly.

What other factors would you add to this list? Why do I bother with these open-ended questions? Because I just like the sound of my own keyboard clicking (yeah, it’s an old school keyboard).

For a more positive spin on load testing efforts, see 7 Steps to Load Testing Bliss.

No Load Testing In Paradise

Friday, May 22nd, 2009 by The Director

Most recruiters don’t have to deal with a job listing crashing under load. This one should have.

The chance to be the caretaker of a tiny tropical island in Australia has sparked so much interest around the world that a rush of applications crashed the website advertising the post.

The job, which offers a salary of $105,000 to spend six months on the Great Barrier Reef island of Hamilton, has been inundated with hundreds of thousands of prospective candidates.

A simple little site, probably not worth load testing at all.

I suspect that the salary is in AUD, not USD, so imagine I’m making my obligatory foreign currency joke here.

Every Year, Somebody’s Web Site Falls Down

Monday, December 1st, 2008 by The Director

This year’s winner in the Which Web Site Was Inadequately Load Tested Before Black Friday Sweepstakes is was inaccessible to U.S. shoppers for two hours on Friday in what was the most notable Web hiccup of the holiday gift-buying season’s official start.

Sadly, also had problems last year, which indicates an institutional problem of some sort, not merely a technical glitch.

Know Your Load, And Expect The Spikes

Friday, May 9th, 2008 by The Director

Also in this month’s Software Test and Performance, Edward J. Correia talks about his experience with a Web site that’s suddenly national in scope, leading to a huge spike in visitors/users:

Watching the Today Show one morning in late January, I saw a segment about, a Web site relatively new at the time that lets people identify and rate their neighbors—good or bad— and post comments about them. The idea is to give people about to relocate a semblance of who lives nearby.

Those media blasts will spike your load far beyond what your normal operating parameters would indicate. The site normally runs 100 concurrent users and has tested out okay at 200 users will get to drink from the firehose after a piece on a morning program.

You’d better keep aware of whatever other marketing or PR is going on with your site to know when to worry.

I once worked on a site with rich media that was getting the full court press, including a Good Morning America segment. You’d better believe that your Director made sure that ship weathered the storm, including rejecting an ill-conceived “optimization” at the end of an all-nighter, werd.

(For a comment on another program that failed, see the post on the Simpsonize Me thing.)

Also, take note in Correia’s column about what application behavior he expects under load, and make it so.

Here’s Your ROI Statement For Load Testing

Monday, April 7th, 2008 by The Director

You don’t want to piss off 100,000 bikers, do you?

The crash Saturday of the ticket-sales Web site for Harley-Davidson’s 105th Anniversary Celebration entertainment – Bruce Springsteen and the E Street Band – left many fans angry, empty-handed and wondering whether concert tickets were still available.

Tickets officially went on sale for the Aug. 30 concert at 8 a.m. Saturday, but the Web site crashed about 30 minutes later. Harley-Davidson reported Saturday evening that ticket sales will resume at 8 a.m. April 14 once the bugs are worked out.

Put that in your risk assessment plan.

Unexpected Server Load = “Malicious Attack”

Tuesday, October 23rd, 2007 by The Director

As you well know by now, the Colorado Rockies server suffered a meltdown when World Series tickets came on sale. Rockies officials claim it was a malicious attack:

Colorado Rockies officials said Monday their computer system for online-only World Series ticket sales was the target of an “external malicious attack,” but online ticket sales were to resume at noon Tuesday.

Team spokesperson Jay Alves couldn’t immediately provide details of the attack.

You know what the attack was? Everyone wanted World Series tickets, and each customer was multiplied because as a group, they overcame limitations of 1 user 1 computer:

McLeod said he has Internet access from his apartment building but thought the library’s computers might be faster. His mother, father, uncle and girlfriend were trying to buy tickets from other computers, he said.

Or some overcame bandwidth exemptions:

Super-fast computers normally used only during emergencies were to be staffed Monday so state employees could buy Rockies World Series tickets online.

Until word of the plan got out, that is.

“I need volunteers to help push buttons in attempting access,” David Holm, recently the acting director of the state Division of Emergency Management, said in an e-mail obtained and released by KUSA-TV on Friday. “You will need to use break time, lunch time or leave time to do this and the only real perk I can offer right now is that if someone does not pay for their tickets within 3 days, you will get first crack at them.”

And here’s a guess: some ticket brokers or perhaps some developers came up with scripts that submitted orders to the Web site to improve their chances of getting tickets.

That’s not a “malicious attack.”

The servers failed from overwhelming demand. Of course, admitting that diminishes the brand, I suppose. But your company got to drink from the fire hose, Colorado Rockies; no one can fault you for failing to predict that your team would ever get to the World Series or what that could do to your systems.

Other stories on the breakdown:

And as a reminder, ungentle reader, your friend and mentor The Director would have found a problem here because he doesn’t run sissy load tests. However, no doubt the project would move forward even with the failure under extreme load because the stakeholders would be comfortable blaming DDOS.

Keeping Your Test Data Clean

Monday, September 24th, 2007 by The Director

There are many things one can learn from  this story:

Symantec Corp.’s early-warning system gave its enterprise customers a brief scare late Friday when it erroneously sent an alert that said an Internet-crippling attack was in progress.

However, the one I want to tease out is this lesson: always keep your junk data clean and readable and clearly defined as test data.


Microsoft Warstrike Load Testing Software

Thursday, September 6th, 2007 by The Director

Well, it’s not really Microsoft Warstrike, it’s Microsoft Web Application Stress Tool (WAST, which as you old Bard’s Tale fans know was the keystroke combination for the Warstrike conjuror’s spell, good for 4-16 points of damage on a group and a pretty potent weapon). Now, Microsoft Web Application Stress Tool lies buried in the Microsoft Downloads section (here). Screenshot:

Microsoft Web Application Stress Tester
Click for full size

It’s an old download (ca. 2002), but it looks to work with IE 7. So if you’re looking for a free rudimentary load-style tester, here’s something.

Sissy Load Tests

Tuesday, July 10th, 2007 by The Director

In this week’s SD Times, Larry O’Brien identifies a problem that load testing didn’t uncover:

Perhaps the only thing worse than a slow uptake of your application is a smash hit. Users have a way of outfoxing everything, including load tests, and the imperative to respond to existing customers can absorb all the working hours of a team that is scheduled to move on to the next version. Worse, when a product is exposed to an order of magnitude more users than planned and when the product is used more intensely than anticipated, the defect list grows rapidly, potentially panicking the team into treating the symptoms, not the causes. The resulting chaos can easily derail a team, especially one new to agile processes, where “the customer is always right” and being responsive were the values that led to the success in the first place.

Not long ago, I witnessed this very problem. I was engaged to work on the requirements and architecture of The Next Phase, which didn’t seem to have a lot to do with The Current Deployment, whose two big features were a comprehensive audit trail for management and a Web-based “dashboard” that gave users a much better view of their own context. Following the principles of “You Ain’t Gonna’ Need It” and “Don’t Repeat Yourself,” the dashboard and the auditing facilities used the same messages to request information; the dashboard, of course, stripped out the huge blobs of auditing data and presented a much-compressed summary. What was not anticipated (note the use of the passive tense to avoid blame) was that the users found the historical perspective of the dashboard very valuable and configured their dashboards to retrieve not just a day or two of history, but often everything they did in the past month.

Listen, there’s “load testing” and then there’s LOAD TESTING, and I would wager this organization did some “load testing.”


wordpress visitors