Sometimes, Automated Testing Is Folly
In the message from the editor in the November 2006 (pdf) issue of Software Test & Performance, Edward J. Correia expresses a basic belief in the magical potency of automated testing:
Repetitive tasks are bad enough—having to perform them repetitively is insane. If something can be automated, it should be. Even if it takes 10 times longer and costs a hundred times more than the original task itself, it pays in the long run to automate your tests.
I cannot tell you how many times I’ve gone into interviews and sales calls for projects where the other person across the table blurts out the benefits of automation and how the company wants to use automation to build a complex suite of automated tests to ensure 100% code coverage to run with nightly builds to ensure project success.
But the automated test evangelists, speaking in their tongues, are wrong. Sometimes automated testing is a bane.
In many cases, I’ve found that the people who want the automation see it as a way to make up for a lack of QA resources. Often, in a tight schedule, the organizations hope that the tests can run overnight and supplant the hours that a tester would need to run the same tests. They have 400 hours total to build a Web application and 80 hours for testing, so they think that one person can come along, evaluate a tool, produce the automated test suite, and spend time in meetings. Automation, they hope, will help solve the problem the timeline and budget create for quality assurance.
Automated testing is an investment that pays off over time, often a long time. If you’re building an application with a months-long release cycle and have an actual quality assurance team that can dedicate a person to doing automated testing, you will see a great deal of benefit in building an automated test suite. The repetitive tasks will be automated, and a basic set of regression tests can run against every build quickly so long as the subsequent builds don’t overhaul the main functionality nightly. The non-automated testers can attack the new functionality first with manual tests, flushing out the defects before the automated scripts are built—and broken—on defective code. Then, when new functionality stabilizes and the functional tests become regression tests, then the automated tester builds the scripts and begins running them nightly.
But in smaller projects where the time or the additional resources aren’t available, automated testing is not the answer. If you’ve got one tester on a project, you should have him spending all of his testing hours on testing, not on building scripts which would account for one-tenth of the number of tests he could run while he’s building the scripts and then have the tester spend fifty percent of his remaining time troubleshooting the scripts and modifying them to make the last minute changes required by the last minute changes made to the code.
Short timelines and limited testing resources don’t lend themselves to a good automated testing program; in fact, wasting time on automated testing actually diminishes the efficacy of QA as the poor, beleaguered tester doesn’t get enough time to run the maximum number of distinct test cases against the application and handle all the time-consuming tasks—that is, reproducing and tracking defects—that go along with any test effort.
If your main purpose in a short project is quality, you’ll use a lot of manual testing and maybe some automated stuff filled in. But if you’re looking to evangelize or dazzle your clients or internal stakeholders, or if you want to expand your budget or pad your resume with a skill set, I guess automated testing will work for you in any case, regardless of the ten times spent on it or the hundredfold increase in cost.
