An experienced automated tester is going to spend more time trying to tell you what automated testing isn’t instead of what it is. Because you’ve probably got a pie-in-the-sky you’re slicing to share for dessert.

Cue Trisherino:

I think people have a tendency to greatly underestimate the difficulty of writing a good GUI-automation suite. It’s not as simple as record and playback, and it’s not like building regular software.

I’ve seen experienced developers underestimate it many times. Inevitably, they end up getting very frustrated and complaining about how rubbish the automation tools are. And yes, the poor quality of automation tools is definitely part of the problem. Developers are used to using very polished tools, lovingly crafted by developers, for developers. Testers are used to using either bloated overpriced commercial tools, condescendingly oversimplified so that “anyone” can use them, or well-meaning but under-maintained free tools, struggling to keep up with the latest technologies.

What she said.

You know why developers are so hot for automated testing? They’re used to thinking they can push software around. Unlike some QA people who refuse to be intimidated by their obvious GENIUSSSS!

Automated tests are suitable for the following purposes:
-    Regression testing for a stable system that will be run on a regular basis
-    Fast data creation in test systems where the database must be wiped on a regular basis

Automated tests are NOT suitable for the following purposes:
-    Testing new functionality – this should be done manually before automated tests are created
-    Regression testing systems that are expected to have significant user interface changes. Large changes to the user interface require a lot of maintenance for automated tests.

You know, testers make these arguments over and over again, but I’ve gone into a number of places to talk about starting QA efforts on major product lines or to work on smaller (160 hour) projects where the principal involved wants automated testing.  Usually on an evolving product and with only one QA person.  Try as I might to dissuade them, they go out and find someone willing to bill them less fruitful hours of QA work because that’s what the client wants.  And the client/employer gets it: an automated effort of some sort, a low defect count (because the QA person spent hours selecting/writing/maintaining automated scripts instead of testing.

But Scary Tester’s and my commentaries fall on sympathetic ears.  Meanwhile, Baseline magazine will run a bunch of ads from software companies selling automated testing software and amid a splashy article about how automated tests can do the work of 20 monkey testers.

I think I’m repeating myself, aren’t I?

Once QA has the new version in hand, they try to run the old regression test suite against the new version of the application. It runs, but an overwhelming number of test case failures are reported because the code has changed so much.

At that point, QA often thinks, “Instead of trying to modify these test cases or test scripts for the new version, we [sic] might as well go ahead and test it by hand because it’s the same amount of work, and even if I update it now, I’ll still have to update it all over again for the next version.” So they [sic] end up testing by hand, and typically come to the conclusion that automation is overrated.

Perhaps he did mean to apply QA was schizophrenic in referring to itself in two different tenses in its own thoughts, neither of which matched the singular antecedent “QA.”

Regardless, his point mirrors one that I alluded to in Sometimes, Automated Testing Is Folly. That is, sometimes the maintenance hit for an automated testing suite outweighs its perceived benefits.

However, Adam Kolawa offers this solution:

One is that team leaders must allocate sufficient budget and time for regression test suite development and maintenance. I’ve found that the best results are achieved when there’s roughly a 50/50 distribution of effort between writing code that represents the functionality of the application and writing code that verifies that functionality. [Emphasis mine]

I’ll pause here until such time as you pick yourself from the floor in shock or in laughter.

Perhaps that is a good mix, and perhaps it does make sense, but good luck on convincing your team stakeholders and pursestring and hourstring holders that you need one QA for every developer.

If you find that promised land of coffee and doughnuts, send me an application.

Why, for instance, do we build software to test other software? This question has never before occurred to me, nor does it parallel such mysteries as people who are financially wealthy but short on values. But it does bear some discussion.

Does he then contemplate the possibility that trusting software to test software is something like telling criminals to police themselves? Nah, he just marvels at the beauty of it. As he should, since the automated software companies are the ones buying the ads in his magazine.

However, we at QAHatesYou.com disagree with his conclusion:

Software is very good at automating things. So when automated testing is the need, why not use the best tool for the job? For the practice of automating software testing, the best tool happens to be more software. Sometimes the best tool is staring you right in the face.

Here at QAHatesYou.com, we have found in our experience that the following are sometimes better solutions, especially when tailored to limited budgets:

• Zombies. All you need are a recurring maintenance budget, i.e., brains. You can certainly find some unused brains on your development team anyway. So raise some dead and show them which keys to push, and wallah! Automated software testing using the undead.
• Steam piston driven software appliances. All you need is a machine shop, some wrenches, and boiling water to build complex steam-driven keyboard punchers. Mouse-handling and pointing-and-clicking are less accurate, so you’ll have to work around that. Also, remember to calibrate the finger-rods correctly, or they will punch right through the keyboard instead of efficiently delivering the keyclick you want.
• Monkeys. Just kidding. We use all our monkeys for new functionality testing.

Automated software testing is really only possible through the use of software, which comes with its own hazards which I’ll go into some other time.