The Turk, the Mechanical Turk or Automaton Chess Player was a fake chess-playing machine constructed in the late 18th century. From 1770 until its destruction by fire in 1854, it was exhibited by various owners as an automaton, though it was exposed in the early 1820s as an elaborate hoax.[1] Constructed and unveiled in 1770 by Wolfgang von Kempelen (1734–1804) to impress the Empress Maria Theresa, the mechanism appeared to be able to play a strong game of chess against a human opponent, as well as perform the knight’s tour, a puzzle that requires the player to move a knight to occupy every square of a chessboard exactly once.

The Turk was in fact a mechanical illusion that allowed a human chess master hiding inside to operate the machine. With a skilled operator, the Turk won most of the games played during its demonstrations around Europe and the Americas for nearly 84 years, playing and defeating many challengers including statesmen such as Napoleon Bonaparte and Benjamin Franklin. Although many had suspected the hidden human operator, the hoax was initially revealed only in the 1820s by the Londoner Robert Willis.[2] The operator(s) within the mechanism during Kempelen’s original tour remains a mystery. When the device was later purchased in 1804 and exhibited by Johann Nepomuk Mälzel, the chess masters who secretly operated it included Johann Allgaier, Boncourt, Aaron Alexandre, William Lewis, Jacques Mouret, and William Schlumberger.

If you’re automation effort requires the active, ongoing effort of someone to keep the fragile scripts built with an unperfect test tool up-to-date with the application, you’re not building an automated test suite–you’re building The Turk.

Granted, to some organizations, any QA is diminishing returns. Fortunately, if you have a job in QA, you don’t work for an organization that believes that. However, at some point, your organization thinks it has had enough QA. As far as testing goes, I argue that you cannot really guess at what point that happens. You might run hundreds of test cases through the third time, perhaps using automation for the routine stuff, and not uncover new defects. But a show-stopping defect might lie outside whatever tests you have mapped out and tried now, but with another week you might have thought up some new wrinkle in workflow to try (or might have arisen after daylight savings time started the week after you stopped testing or some other environmental concern).

So I won’t concede at any point that there’s a diminishing return at any point at the end of the test cycle. However, I do think the law applies at the beginning of the cycle with automated testing.

I say this after reading this:

In my last blog I shared the “World Quality Report” from Capgemini. One of the QA challenges that the report cited was a slow adoption of automation tools early in the cycle. Why? According to the report, it’s hard to realize the ROI.

The author goes on to give an example of how automation helped realize ROI over the long term, particularly when the product reached a level of stability and regression tests were run over and over again. That is when it is the proper time to introduce automation, but it is not early in the cycle.

Early in the software development cycle includes requirements gathering, discussions with users, some sort of mockups, and early, unstable builds that are not feature complete and whose GUIs will change based on feedback and whatnot. In these early stages, it does not make sense to add an automated tester except to bring up concerns and points about automated testing, including how to best build the application to include testable GUI practices and test harnesses and whatnot.

Starting to build automated scripts against mockups or against those unstable, buggy builds would require more effort in script maintenance than you would receive benefits. If you need to select a tool, it’s probably too early to make the decisions as to which tool(s) to use since the GUI might go in a different direction that would make a different tool (or tools) a better fit.

The law of diminishing returns definitely applies to automated testing early in the SDLC.

On the other hand, if your product/project is going to run long term with numerous build cycles, it does make sense to automate. On the back end.

Automated testing, really, is a misnomer. The testing is not automated. It requires someone to build it and to make sure that the scripts remain synched to a GUI. If the GUI changes or the user workflow changes, a tester needs to not expend effort on testing, but on revising the automated scripts–which might run once for a single build cycle before requiring further revision.

It’s computer-run scripted testing. If we just called it that, people would understand it better and expect less magic from it.

JPMorgan Chase is trying to move past three days of problems on its online banking site with an apology and an explanation that seems to put the cause on a third party.

The bank’s online site went offline Monday night and remained offline Tuesday. Service appeared restored by Wednesday, although there were some reports by Twitter users of problems.

The bank, in a statement posted online, said it was “sorry for the difficulties” that customers encountered, and said “we apologize for not communicating better with you during this issue.”

At first, Chase simply cited a “technical issue” for the problem. It has since provided a little more information.

The bank, the nation’s second largest, said in a separate statement that a “third party database company’s software caused a corruption of systems information disabling our ability to process customer log-ins to chase.com.” It added that the problem “resulted in a long recovery process.”

Now, how can you try to keep this from happening to you?

• Compel your vendors to tell you about their updates. Ideally, you would get a chance to test your software against their new versions before they promote them to production, but at the very least, they better tell you when they plan to put things up so you can test immediately. Remember, your “trusted” partners are organization filled with the same lying developer dogs as yours, but without the QA.

• Don’t do business with companies that practice continuous deployment. Seriously, they can promote at will and at whim, so your mission-critical software can fail at any time, without any warning, and without any clue that it’s not your fault.
• Run automated smoke tests against your production site as often as you can stand. Depending upon the nature of the application, this might only be daily, but the more frequently you can sanity check your production environment, the better. There’s nothing better than calling your head of development on Christmas Eve to tell him the site’s down before your users or clients even know.

Remember, you have no trusted partners. You should trust them even less than you trust your own organization, if you can imagine that.

An experienced automated tester is going to spend more time trying to tell you what automated testing isn’t instead of what it is. Because you’ve probably got a pie-in-the-sky you’re slicing to share for dessert.

Cue Trisherino:

I think people have a tendency to greatly underestimate the difficulty of writing a good GUI-automation suite. It’s not as simple as record and playback, and it’s not like building regular software.

I’ve seen experienced developers underestimate it many times. Inevitably, they end up getting very frustrated and complaining about how rubbish the automation tools are. And yes, the poor quality of automation tools is definitely part of the problem. Developers are used to using very polished tools, lovingly crafted by developers, for developers. Testers are used to using either bloated overpriced commercial tools, condescendingly oversimplified so that “anyone” can use them, or well-meaning but under-maintained free tools, struggling to keep up with the latest technologies.

What she said.

You know why developers are so hot for automated testing? They’re used to thinking they can push software around. Unlike some QA people who refuse to be intimidated by their obvious GENIUSSSS!

Automated tests are suitable for the following purposes:
-    Regression testing for a stable system that will be run on a regular basis
-    Fast data creation in test systems where the database must be wiped on a regular basis

Automated tests are NOT suitable for the following purposes:
-    Testing new functionality – this should be done manually before automated tests are created
-    Regression testing systems that are expected to have significant user interface changes. Large changes to the user interface require a lot of maintenance for automated tests.

You know, testers make these arguments over and over again, but I’ve gone into a number of places to talk about starting QA efforts on major product lines or to work on smaller (160 hour) projects where the principal involved wants automated testing.  Usually on an evolving product and with only one QA person.  Try as I might to dissuade them, they go out and find someone willing to bill them less fruitful hours of QA work because that’s what the client wants.  And the client/employer gets it: an automated effort of some sort, a low defect count (because the QA person spent hours selecting/writing/maintaining automated scripts instead of testing.

But Scary Tester’s and my commentaries fall on sympathetic ears.  Meanwhile, Baseline magazine will run a bunch of ads from software companies selling automated testing software and amid a splashy article about how automated tests can do the work of 20 monkey testers.

I think I’m repeating myself, aren’t I?

Once QA has the new version in hand, they try to run the old regression test suite against the new version of the application. It runs, but an overwhelming number of test case failures are reported because the code has changed so much.

At that point, QA often thinks, “Instead of trying to modify these test cases or test scripts for the new version, we [sic] might as well go ahead and test it by hand because it’s the same amount of work, and even if I update it now, I’ll still have to update it all over again for the next version.” So they [sic] end up testing by hand, and typically come to the conclusion that automation is overrated.

Perhaps he did mean to apply QA was schizophrenic in referring to itself in two different tenses in its own thoughts, neither of which matched the singular antecedent “QA.”

Regardless, his point mirrors one that I alluded to in Sometimes, Automated Testing Is Folly. That is, sometimes the maintenance hit for an automated testing suite outweighs its perceived benefits.

However, Adam Kolawa offers this solution:

One is that team leaders must allocate sufficient budget and time for regression test suite development and maintenance. I’ve found that the best results are achieved when there’s roughly a 50/50 distribution of effort between writing code that represents the functionality of the application and writing code that verifies that functionality. [Emphasis mine]

I’ll pause here until such time as you pick yourself from the floor in shock or in laughter.

Perhaps that is a good mix, and perhaps it does make sense, but good luck on convincing your team stakeholders and pursestring and hourstring holders that you need one QA for every developer.

If you find that promised land of coffee and doughnuts, send me an application.

Why, for instance, do we build software to test other software? This question has never before occurred to me, nor does it parallel such mysteries as people who are financially wealthy but short on values. But it does bear some discussion.

Does he then contemplate the possibility that trusting software to test software is something like telling criminals to police themselves? Nah, he just marvels at the beauty of it. As he should, since the automated software companies are the ones buying the ads in his magazine.

However, we at QAHatesYou.com disagree with his conclusion:

Software is very good at automating things. So when automated testing is the need, why not use the best tool for the job? For the practice of automating software testing, the best tool happens to be more software. Sometimes the best tool is staring you right in the face.

Here at QAHatesYou.com, we have found in our experience that the following are sometimes better solutions, especially when tailored to limited budgets:

• Zombies. All you need are a recurring maintenance budget, i.e., brains. You can certainly find some unused brains on your development team anyway. So raise some dead and show them which keys to push, and wallah! Automated software testing using the undead.
• Steam piston driven software appliances. All you need is a machine shop, some wrenches, and boiling water to build complex steam-driven keyboard punchers. Mouse-handling and pointing-and-clicking are less accurate, so you’ll have to work around that. Also, remember to calibrate the finger-rods correctly, or they will punch right through the keyboard instead of efficiently delivering the keyclick you want.
• Monkeys. Just kidding. We use all our monkeys for new functionality testing.

Automated software testing is really only possible through the use of software, which comes with its own hazards which I’ll go into some other time.