Archive for May, 2011

The Three Types of Applications

Tuesday, May 31st, 2011 by The Director

The 1982 video game Sinistar had an interesting bug:

Sinistar contains a bug that grants the player many lives (ships). It happens only if the player is down to one life and Sinistar is about to eat the player’s ship. If a warrior ship shoots and destroys the ship at this moment, it immediately takes the player to zero lives, and Sinistar eating the player subtracts another life. Since the number of lives is stored in the game as an 8 bit unsigned integer, the subtraction from zero will cause the integer to wrap around to the largest value representable with 8 bits, which is 255 in decimal.

I have never tested video games. Frankly, there are three types of applications:

  • Do applications, which are designed to have one task or set of tasks, and the outcome is known. The workflows are pretty closed and known.
  • Create applications, which are designed to let a user make something from scratch. A lot of rules are in place as to how the user creates, but the workflows and end results are open-ended.
  • Video games, where there are also some rules and some predicted end results, but the users can get there in myriad ways (or will do something other than those intended end results). Additionally, users will sometimes try to break the established rules just to see what will happen.

Frankly, I’ve done most of my testing in the context of the first. Video game companies rely heavily on beta testing and crowd sourcing to get the greatest number of monkeys at the typewriters as possible. Outside of the first type, though, you really have to have some creativity, insight, and awareness outside of the normal test case way of thinking because there really aren’t enough test cases to cover everything.

QA Music: Old Timey Rock and Roll

Monday, May 30th, 2011 by The Director

Make your week epic with “O Fortuna” from Carmina Baruna:

Funny, but I don’t remember this actually playing when Shirley’s boyfriend entered the apartment, but it rocks anyway.

Lady Gaga, Accidental Load Tester

Wednesday, May 25th, 2011 by The Director

Lady Gaga performs “Load Tested This Way“: Inc.’s one-day, 99-cent promotion of Lady Gaga’s highly anticipated second studio album, “Born This Way,” is resulting in downloading delays on the Internet retailer’s website due to high volume, the company said Monday.

That, my friends, is why you should always load test to the maximum, not just to the comfortable. Because an event that brings down your site or service is not going to fit in the comfortable range.

The Law of Diminishing ROI

Tuesday, May 24th, 2011 by The Director

Some of you might be familiar with a particular facet of the laws of economics called “The Law of Diminishing Returns.” Basically, it says that after some point, more effort is not going to provide enough benefit/profit to justify the continued effort.

Granted, to some organizations, any QA is diminishing returns. Fortunately, if you have a job in QA, you don’t work for an organization that believes that. However, at some point, your organization thinks it has had enough QA. As far as testing goes, I argue that you cannot really guess at what point that happens. You might run hundreds of test cases through the third time, perhaps using automation for the routine stuff, and not uncover new defects. But a show-stopping defect might lie outside whatever tests you have mapped out and tried now, but with another week you might have thought up some new wrinkle in workflow to try (or might have arisen after daylight savings time started the week after you stopped testing or some other environmental concern).

So I won’t concede at any point that there’s a diminishing return at any point at the end of the test cycle. However, I do think the law applies at the beginning of the cycle with automated testing.

I say this after reading this:

In my last blog I shared the “World Quality Report” from Capgemini. One of the QA challenges that the report cited was a slow adoption of automation tools early in the cycle. Why? According to the report, it’s hard to realize the ROI.

The author goes on to give an example of how automation helped realize ROI over the long term, particularly when the product reached a level of stability and regression tests were run over and over again. That is when it is the proper time to introduce automation, but it is not early in the cycle.

Early in the software development cycle includes requirements gathering, discussions with users, some sort of mockups, and early, unstable builds that are not feature complete and whose GUIs will change based on feedback and whatnot. In these early stages, it does not make sense to add an automated tester except to bring up concerns and points about automated testing, including how to best build the application to include testable GUI practices and test harnesses and whatnot.

Starting to build automated scripts against mockups or against those unstable, buggy builds would require more effort in script maintenance than you would receive benefits. If you need to select a tool, it’s probably too early to make the decisions as to which tool(s) to use since the GUI might go in a different direction that would make a different tool (or tools) a better fit.

The law of diminishing returns definitely applies to automated testing early in the SDLC.

On the other hand, if your product/project is going to run long term with numerous build cycles, it does make sense to automate. On the back end.

Automated testing, really, is a misnomer. The testing is not automated. It requires someone to build it and to make sure that the scripts remain synched to a GUI. If the GUI changes or the user workflow changes, a tester needs to not expend effort on testing, but on revising the automated scripts–which might run once for a single build cycle before requiring further revision.

It’s computer-run scripted testing. If we just called it that, people would understand it better and expect less magic from it.

QA Music: See You In Hell

Monday, May 23rd, 2011 by The Director

A British band called Grim Reaper:

I Would Be Remiss If I Did Not Mention

Tuesday, May 17th, 2011 by The Director

My novel is now available:

John Donnelly's Gold cover

The book describes what happens when four IT workers get laid off from their Web travel company and decide to steal a gold bar that their CEO bought in a moment of largesse. I wrote it when I was working at a startup and characterized it at work as “a crime spree perpetuated by laid off employees against their bosses” to protect my own job security.

The book’s Web site is here. It’s currently available as a trade paperback and electronic download from, but iPad and Kindle editions are forthcoming.

Also, if anyone could suggest a good free defect tracker for me to put up at the book’s Web site, let me know. As some of you might have gauged from my Twitter posts, I had a hard time ending the rounds of copyediting. Every time I ran through the book, I found some other error. Weeks with 8 days, semiautomatic pistols turning into revolvers, alma maters changing. It’s enough to drive a QA guy crazy.

So if I get a good defect tracker installed, I’m going to let readers log defects for correction in a future edition.

But go, buy it, read it, and prepare your lines of attack.

QA Music – Street QAing Man

Monday, May 16th, 2011 by The Director

Lynch Mob.

QA Music: The Road to QA

Monday, May 9th, 2011 by The Director

Sorry, I meant “The Road To Nowhere”:

Two Minute QAte: Target: Querystring

Thursday, May 5th, 2011 by The Director

Here’s a shorter Two Minute QAte discussing the querystring as a vector for bad data and bad things into your application:

“Remember the querytring!”

QA Music: Your Project Manager Pleads

Monday, May 2nd, 2011 by The Director

So, it’s Monday morning. Time for a status meeting. “Are things on schedule?” your project manager asks.

As a public service message, allow me to translate for you:


wordpress visitors