QA Hates You

I’ve said it’s dangerous to base your Web browser compatibility testing matrix on a press release.

But this story might have some use to you:

Google’s Chrome edged past Microsoft’s Internet Explorer (IE) last week to become the world’s most widely used browser, according to data from an Irish metric firm.

Chrome’s average usage share for the week of May 14-20 was 32.8%, said StatCounter, an analytics company that tracks browser and operating system trends. For the same week, IE’s share was 31.9%.

If you read the whole story and not just the headline, you’ll find that this metrics-providing firm used some data modeling to conclude as it did, and that other firms with other ideas about data models continue to come up with different results.

However, you can learn something from this:

• It’s important to continually re-analyze your assumptions.
  If you thought it was important to test a browser last year, you might need to change your Web testing to accommodate the changing realities. I acknowledge this so much that I’m no longer mentioning testing in Netscape or AOL Explorer even though I still have those browsers installed in the lab.
   
• Just because it’s a cool browser doesn’t mean you shouldn’t test in it. Or, more to the point, because it’s an IE and Firefox world in the popular culture consumer mindset world (in the popular culture, the world runs Safari. Inspect every television program, commercial, or print advertisement showing a Web page, and 97% of the time, you’ll see the Safari browser window around it, or I’m not a guy with an English degree just making statistics up). More to the point, it’s important to remember that sometimes you do need to test using the things your designers and developers think is cool. They’re not always wrong, just mostly.

SD Times magazine has it in for testing in the May 2012 issue.

First, in the article on Kanban (“Kanban: Is It In The Cards?“), when it comes time to illustrate a blocked task, we get a likely scapegoat:

Secondly, we get an Industry Watch column “Testing culture undergoes dramatic shift” that describes how testing is becoming more relevant by adapting itself to what the developers want or by turning into tech support people.

This culture shift in testing might help testers shed the image of being impediments to software releases instead of facilitators of quality software releases. “How do we work with the development team in this brave new world?” Sterling asked. “The role of testers in the cloud becomes a huge value-add. Testers triage the feedback data, and turn around to tell developers, ‘Your customers want features 5, 10 and 43.’

“All of a sudden,” he said, “your success in getting your next check is pinned on testers getting you the feedback data to deliver what those customers want.”

I agree with some of the sentiments in the article, but I’d like to say that it’s not that testing is changing to fit the needs of software development. I’d like to think quality assurance professionals are helping to change the whole kit-and-kaboodle to provide better quality.

As long as we’re actually making the developers do what we want and making them think it’s their idea, though, we’re golden.

XKCD uncovers a bug that QA should always find:

(Thanks to the most beautiful developer I know.)

Apparently, they’re on the Blizzard servers:

The Diablo 3 servers are at full capacity, preventing many from playing the game.

Players across the globe are reporting “Error 37″ when trying to log in following Diablo 3′s midnight launch in the UK at 11pm last night and, just hours ago, on the West Coast.

“Due to high concurrency the login servers are currently at full capacity,” Blizzard wrote on the Battle.net forum. “This may cause delays in the login process, account pages and web services.

The best part, or worst part, depending upon whether you’re a mere observer or a customer who plunked down $60 for the game: Blizzard actually warned they weren’t going to have enough server capacity to handle their user needs in a blog post last week. And didn’t accommodate the usage spike until it happened.

(Seen via Fred Beringer tweet. I’m not a fan of the video game series. It reminds me too much of my day-to-day work.)

Here’s a book excerpt in the Wall Street Journal on improving your judgment of risk:

Most of us have to estimate probabilities every day. Whether as a trader betting on the price of a stock, a lawyer gauging a witness’s reliability or a doctor pondering the accuracy of a diagnosis, we spend much of our time—consciously or not—guessing about the future based on incomplete information. Unfortunately, decades of research indicate that humans are not very good at this. Most of us, for example, tend to vastly overestimate our chances of winning the lottery, while similarly underestimating the chances that we will get divorced.

Psychologists have tended to assume that such biases are universal and virtually impossible to avoid. But certain groups of people—such as meteorologists and professional gamblers—have managed to overcome these biases and are thus able to estimate probabilities much more accurately than the rest of us. Are they doing something the rest of us can learn? Can we improve our risk intelligence?

Sarah Lichtenstein, an expert in the field of decision science, points to several characteristics of groups that exhibit high intelligence with respect to risk. First, they tend to be comfortable assigning numerical probabilities to possible outcomes. Starting in 1965, for instance, U.S. National Weather Service forecasters have been required to say not just whether or not it will rain the next day, but how likely they think it is in percentage terms. Sure enough, when researchers measured the risk intelligence of American forecasters a decade later, they found that it ranked among the highest ever recorded, according to a study in the Journal of the Royal Statistical Society.

The excerpt says that you can improve your risk analysis abilities by getting immediate feedback. However, if you’re trying to answer the risk of deploying undertested software with the potential for hidden defects or if you’re estimating the chances of a discovered error occurring in the wild, that feedback might not be immediately available if the circumstances don’t occur until six months after the software is in use.

At any rate, it’s an article worth reviewing and maybe it’s worth getting the whole book Risk Intelligence: How to Live with Uncertainty.

George Takei shared this photograph on Facebook:

Class, who can tell me what’s wrong with this picture?

A Non Sequitor cartoon from April 9, 2012:

Strangely enough, QA does just that.

And, yeah, I am a month behind on the local newspaper. I’m even further behind on the Wall Street Journal, which means when I try to catch up on them, it’s almost like living as Time in Piers Anthony’s Incarnations of Immortality series.

Appearing in the new ST & QA Magazine, it’s “When Users Collide“. (Registration required.)

Computerworld reports IE ‘silent’ upgrade helps put newest browser on Windows: Stats show some Windows 7 and Vista users upgraded to IE9, but the new practice affected few XP users:

Microsoft’s decision late last year to switch on “silent” upgrades for Internet Explorer (IE) has moved some Windows users to newer versions, but has had little, if any, impact on the oldest editions, IE6 and IE7, according to usage statistics.

Being in QA means you get to say “I Told You So” an awful lot. But it never gets old.

Hidden in this Forbes article about Amazon.com is a disturbing metric, particularly disturbing if you consider it in any detail. The metric:

Even the tiniest delay in loading a Web page isn’t trivial. Amazon has metrics showing that a 0.1 second delay in page rendering can translate into a 1% drop in customer activity.

Why is this particularly disturbing? Customers go to Amazon to buy. What is that slow page low time doing to your site’s visitors whose attachment and commitment to your site might be much lower?

By the way, you are doing your performance testing from outside the corporate network to get a feel for the load times on the actual Internet, aren’t you? I’d feel a little silly asking it, except I am a seasoned QA consultant. You might not be.

I realize I’m a little late to the annual celebration of a maritime disaster, but back when it was timely last week, Popular Mechanics did a piece called Why We’re Still Learning the Lessons of Titanic about how even the most up-to-date engineering can fail catastrophically. A taste:

In one respect, little has changed. As the recent loss of the Italian cruise ship Costa Concordia demonstrates, bad decision making can overcome even robust engineering. Virtually all man-made disasters—including the Three Mile Island nuclear accident, the space shuttle Challenger explosion, and the BP oil spill—can be traced to the same human failings that doomed Titanic. After 100 years, we must still remember—and, too often, relearn—the grim lessons of that night.

No disaster is a single event. Complex systems rarely fail without warning. Instead, accidents are the product of decisions made over hours, days, and sometimes years. Those choices are shaped both by the culture of the organization—whether it’s NASA or the White Star Line, which owned Titanic—and by outside pressures.

When you’re mapping out, building, and testing applications, remember the human failure element. Remember the peril of the badmins.

And when you’re doing your risk analysis about whether critical-but-unlikely bugs need to be fixed or what extreme conditions you should test for, you and everyone else needs to remember that unlikely is not impossible, but catastrophic is always catastrophic.

Take a look at this:

As I mentioned yesterday, this is a 24-volt double pole contactor for an air conditioning unit.

But it’s also akin to a computer program. It’s a simple mechanical switch, but it functions like any method or function in an application. It takes input from the main circuit board, a 24 volt current that triggers the switch. When the switch is triggered, the condenser and compressor come on and make the magic happen. When the current stops, the switch opens and the external unit shuts down.

So. What can happen to impede this process? How would you test it?

Of course, one would send a 23 volt current along to see if it triggered. And then 25. And then, if one was really sadistic, one would pipe the electric version of Hamlet at it (which would explain why the lights in the building just dimmed).

What happens if the switch sticks in the partially on position? What happens if current travels between the contacts incorrectly (a shower of sparks and a bad switch, which explains why this one is on my desk and not in my air conditioning unit). In many cases, the problems you would find would be the result of installation issues, that is, deployment of this object. That’s when the wires can get hooked to the wrong screws and whatnot. What happens then? Hopefully, just a failure of the device and not a cascading catastrophe.

When you’re looking at an application, try visualizing it as a physical object with access points and egresses. What happens if you cross the streams? Is it only partial protonic reversal? If so, log a defect.

Take a look at this:

This is a 24 volt double-pole contactor for an air conditioning unit. Part number CONT1P030024V.

What does that mean?

A former client of a former client manufactured this sort of thing, so it wasn’t uncommon to see part numbers rolling up the screens during testing and documentation. Each of those lines of data referred to actual objects. Not programming language objects, but physical devices that the software’s users and customers depended on. Those customers probably bought them and installed them in residential and industrial settings to keep people cool during the summer months.

When you’re stuck in a cubicle, office, or testing lab ten hours a day (or thirty-six if you’re g33klady), those little lines on the computer screen are just that.

But out in the real world, they’re real things. Remember that.

A floating survey doesn’t like the elderly:

That’s a pretty strange business requirement. If you get something that looks arbitrary, you do know to challenge it, right?

Make sure there’s some reason that your organization doesn’t want to collect information on someone using an iPad from his or her assisted living facility.

IGN has a couple pages about what it’s like to be a QA tester in the video game industry. The lede:

What gamer hasn’t dreamed about playing games for a living? While this may seem like a great career, and a cool way to get a first job in the games business, the truth is less appealing.

Although the testers cited in the article grouse a lot about the things that make any employer a bad employer from a testing employee’s (or contractor’s) perspective, one element is even worse: The pay is crap.

Because, face it, it’s the kind of job that naive cannon fodder will throw itself at for the very reason listed in the lede: Testing games all day sounds like a lot of fun until you realize that it’s the same tedium as running the same test cases against a piece of database connector software except with better graphics. As long as that fantasy lives, kids will come along and accept that ten dollar an hour wage to do it.

It’s kind of like writing Internet humor. You think it sounds like fun or it might be a springboard to something else, but mostly it’s what it is and leads to more of the same if you’re lucky.

So although I’m not against working on games, it’s for my rate of pay, not a sweatshop’s.

While I was renewing my subscription to Information Week, the survey presented me this question:

The question is:

Which of the following software products, services and/or technologies do you currently or plan to approve, specify, recommend, purchase, or influence the purchase of?

And one of the options is:

Spyware / Malware

By checking this, you do realize that logically you’re saying that your organization uses spyware or malware, right?

So I got a junk text the other day, offering me some Nigerian prince’s Starbucks free sample or something. However, at the end of the text, I spied a bit of the wizard behind the curtain:

If you’ve got a program that spits out text messages, much like a program that spits out email communications, you do check that text message or email, don’t you? Or printed pages from your application?

Because your user might see all forms of output, you need to make sure they’re not embarrassing.

As some of you might know, I’ve been having a little trouble with my DSL connection starting sometime last winter. AT&T operators and technicians have been very polite, for the most part, but they didn’t fix the problem yet.

Additionally, I needed a static IP address to make connections to client networks easier, and this did not go well, either, since most AT&T phone representatives only want to sell U-Verse and transfer you to tech support if you even mention static IPs. I finally got it, but at a bill rate three times what they sell it to business customers.

I sent a letter to the head of AT&T Business and Home Solutions: (more…)

You can find a gallery of pictures from this week’s STPCon here.

Startups Aim to Make Coding Fun: The companies use video-game tricks to make people forget that they’re learning.

For Jacob Arriola, a business development manager for a Spanish media company in Los Angeles, learning to program wasn’t a necessity. But figuring it might help with his job, he started using an online code-tutorial service called Treehouse in January.

After three months with the paid service, he’s earned several dozen badges for completing programming quizzes and challenges, and watching coding-related video lessons. More importantly, he’s built his own website from scratch and made some simple changes to websites that his company runs. “I’m able to do it myself, which is pretty cool,” he says.

You know, QA has had something like for over 30 years now. (more…)