Look For What’s Not There

Let’s all play Abraham Ward:

During WWII, Hungarian-born mathematician Abraham Wald undertook a study with the British Air Ministry to use statistical analysis to help protect bombers flying over enemy territory. The data to be crunched included the number and location of bullet holes on returning aircraft, and the goal was to use this information to determine where to best add armor to the plane’s structure.

A nifty little chart was created to show where the maximum number of bullet holes were located on returning aircraft. This chart showed the greatest damage not on the main wing and tail spars, engines, and core fuselage areas, but rather on the aircraft extremities. Based on this, the Air Ministry suggested adding armor to those extremities. Wald suggested they were dead wrong.

Wald said more armor should go on the places that had the least holes. Huh? What was he thinking?

Wald was keeping the Air Ministry from falling into the “survivorship bias”: they were forgetting the their data did not include the planes that had been lost. If the returning planes had no holes in their wing spars and engines, the better assumption to make is that even a few holes in those places were deadly: no damage was recorded in those areas because those planes were the ones that had crashed. Wald recommended more armor in those data-free areas.

That’s a good lesson for testing and quality assurance. Apply them thusly:

  • Your requirements lack certain conditions and workflows to account for. You can hope users will follow your happy paths, but what if they do not?
  • If certain elements of the application are not yielding defects, you’re not testing them enough. You’ll find something to complain about everywhere if you look.
  • If your customers aren’t calling the help desk with questions or problems about certain features, they’re not using them. Can you drop some? Seriously, you want to give every user an avatar that only shows on login and on the edit profile page. Why do you hate humanity?
  • If certain people don’t play foosball with QA, what, are they some kind of continental foosball snobs afraid of three men on the goalie rod and unable to handle the patented QA bank-off-of-the-side-of-the-rightmost-man-on-the-goalie-rod slop shot?

(Note this story did appear on SQA Forums, but I saw it elsewhere since I don’t loiter over there much these days.)

Comments are closed.

wordpress visitors