QA Hates You

You know, reading horror books doesn’t keep me from sleeping like a baby (a colicky baby) at night. What do I read to give me chills and to keep me awake in the darkness, staring at the ceiling and contemplating dark things that might snatch my life away? Books like this book.

The subtitle of the book is And Over 100 Other Stories of Embarrassing and Funny Stories of Technology Gone Mad. It collects a number of humorous incidents where software or software-related processes have gone awry and made the papers, causing great embarrassment for the companies responsible. I wouldn’t call them epic fails, because in the 21st century, epic fails are fleeting. These are legendary failures still half-remembered and fully documented for posterity.

Reading through the book, one identifies some areas of risk to pay particular attention to if you’re trying to prevent your company’s failures from becoming the stuff of legend and Snopes articles. These include:

• Preventing the leak of test data.
  In many of the stories, great fun happens when test data or placeholder material goes into production. Such as the titular “Dear customer, You are a loser.” email or the “Rich Bastard” test name in a mail merge. When you’re creating test data, don’t be clever or wry, since that might leak out. Play it straight. And for Pete’s sake, figure out how to purge it before it gets out there.
   
• Review your CMS procedures.
  A lot of the news-stories-that-aren’t-real tales in this book come from instances where content authors somehow put their incomplete works into draft and they end up live on the Web site. This might be because the content management system has issues, or it might be because the content author has the ability to publish his or her own work and inadvertantly does so before the proper time. Sometimes, this happens without a CMS where code roll-ups get promoted with draft content. Regardless, you need to scrutinize those procedures to minimize the chances of this happening. As a bonus, one of the stories is about a journalist whose story gets promoted to the live site with disrespectful placeholders within it. Have I mentioned that’s a bad thing?
   
• Not understanding practices of the users and building in problems through ignorance.
  Then there’s the story about the guy who got a license plate that said NO PLATE and ended up getting the tickets for every car in the state without a license plate. Or at least those where the police officers had written No plate for a license plate number. If you don’t know what your users’ habits are, you can walk right into a problem where their habits conflict with your software’s interface and abilities.
   
• The impossible calculated numbers.
  The book is rife with the stories of impossible calculation results, such as the trillions of dollars in library fines, the bajillions of dollars in water meter charges, and so on. Does your software have a sanity check to flag outlying calculation results? If not, why not?

This book has a lot for the software quality professional to learn. It exposes patterns of failure we need to recognize and to account for in our testing and rolls up a whole lot of lessons learned meetings into a very browseable 300 or so pages.

Definitely recommended.

Books mentioned in this review:

This book is the autobiography of Richard Marcinko, the man who organized SEAL Team Six. It recounts his history from his days as a lowly enlisted man in the United States Navy and his rise through the ranks as he becomes an officer, a leader, and commander.

So what? you might ask. I’ve read a bunch of books about testing, I’ve read a couple of books about managing, but I’ve never really read a book that captures a career path that mimics the one in software testing.

You start out bucking the system, complaining about the Man, and fighting hard to get quality. Eventually, if you get promoted to team lead and beyond, you have to subvert that fighting instinct to recognize the new environment and to work within its limitations to further your goals and missions.

Marcinko goes from a platoon leader who goes around the leadership he doesn’t like to being a ramrod straight commander of a SEAL team. He was to play the political game a bit and know how things are done in the Navy. When he gets an opportunity, he gets to form his own team, SEAL Team Six, in his own vision–which is closer to snake-eating SEAL grunt than ship-driver (that is, a commander that comes from a different milieu). Marcinko ends up training with his men even when he’s a high-level executive.

Definitely some interesting lessons in this book, but as it is his autobiography and it’s written from the point-of-view of a long time military man, the language is pretty vulgar and the outlook a bit crude. However, Marcinko has written two books that have his picture on the cover in suits and probably don’t have quite as many f-bombs: Leadership Secrets of the Rogue Warrior and The Rogue Warrior’s Strategy for Success.

Or, if you want some more sedate reading, you could pick a book off of James Bach’s recent list, but there’s probably not as much shooting in them.

Books mentioned in this review:

It would be a facile take on this short motivational book to try to explain that QA should look at everything in this book and to do the opposite. This book is all about jumping into a project of some sort and how to best thwart the resistance that will come up when you try to complete it. The book focuses on creating a work of art, a book, a musical compostion, or whatnot.

Unfortunately, in the hands of an evangelist or a developer, he or she will think computer software falls directly into this realm, but it does not.

The tenets of the book include to slop out something before overthinking or before rational thought (seriously) overtakes you. How often have we seen software written like that? Every day? Several times a day? When we sleep fitfully because developers are effectively quashing resistance (that is, anyone who would say that it’s not good enough. Like us.) Begin before you’re ready. Get to work, so to speak.

While that might work for a symphony where the trumpet player blowing a flat note because there’s a misprint on the score won’t bother anyone but the purists or where a typo in a novel is going to cause a little tittering amongst the grammatically aware crowd but won’t derail the narrative, probably. But computer software is not like these things. Computer software, at best, is more akin to nonfiction than a sonnet. Mistakes will impact users in more than superficial ways.

Sure, the book does nod that sometimes you have to evaluate what you’ve learned to correct what you’ve done, but the main impetus of the book is charging recklessly forward. Kind of like when they promise to get some bug fixes into the next sprint, but at the next sprint planning meeting, they are eager to slop new feature out before overthinking or before rational thought overtakes them so that they’re always getting to work and not fixing stuff that’s broken.

I think the book probably does capture the creative mindset and motivates it. Unfortunately, it doesn’t necessarily mean that something of quality will result. But if you read it, maybe it’ll get you started on some project you’ve put off, or maybe it will just give you some resistance to the first-to-market-users-will-forgive mindset that leads to something like Circle more often than it leads to something like Microsoft.

Don’t you read it and go all soft on me now, QA professionals.

And before you ask, yes, this was given to me by a developer.

Books mentioned in this review:

If you’re looking to improve your management chops, emphasis on the chops, you can take some lessons from one of the legendary leaders of history: Attila the Hun.

I recently completed Attila: King of the Huns: The Man and the Myth, and that book offers these lessons from the reign of the warrior-king:

• Build a team of diverse talents.

  As a ruler, Attila was an originator, in some respects a revolutionary. He clearly perceived that if the Huns were to become a great power, as he intended them to be, they must learn from other more advanced peoples. His own intimate circle of advisers, in consequence, consisted largely of foreigners.

  If you hire and promote only people who’ve worked in your business line or only in your particular methodology, you’re going to be bound to only insights and knowledge of those. If you hire from outside them, though, the foreigners might teach you something about what you’re doing that someone of your tribe would not innovate.
   

• Let the farmers farm.
  

  When the Huns occupied a new territory large numbers of people fled before them, but the numbers of those who remained were larger still. Many of those who stayed were agriculturalists, and the Huns not only allowed them to continue to till the land, but encouraged them to do so.

  When you take on a new assignment or position, some turnover might occur as people leave to follow their mentors or because they feel that they, not you, should be the barbarian king of the tribe. Accept that. Those who stay have a lot of experience in the field. Don’t immediately overturn everything they do to conform to your new methods which might be informed and learned, but might not apply. Let those workers show you how they do things, and you can learn from them and, if appropriate, gradually increase the taxes introduce your improvements. Otherwise, more will fly, and you’ll lost a lot of knowledge with them.
   

• Avoid the trappings of the title.
  

  ‘While sumptuous food, served on silver plates,’ he [Priscus] wrote, ‘had been prepared for the other barbarians and for us, for Attila there was nothing but meat on a wooden platter. He showed himself temperate in all other ways, too, for gold and silver goblets were offered to the men at the feast, but his mug was of wood.’

  In dress and general appearance also Attila was noticeably different from the others present. ‘His dress was plain, having care for nothing other than to be clean, nor was the sword by his side, nor the clasps of his barbarian boots, nor the bridle of his horse, like those of the Scythians, adorned with gold or gems or anything of high price.’

  If you’re too busy showing everyone you’re the leader, you’re not leading, you’re pillaging. Back when I was an actual Director of Quality and not merely a Director Emeritus, I had my cards say simply, “Quality Assurance.” Because my title wasn’t important, what I did was important. Plus, you get the respect of the blue collar grunts if you do like they do and live like they do. Or at least you get my respect.
   

• Break something early.
  

  The wholesale destruction of a major town early in a campaign was a strategy adopted by Attila in both France and Germany. Metz recovered to become a city of importance. Aquileia did not. But the news of what happened in both places spread, as it was no doubt intended to, and the rulers of other cities duly took note.

  This is not to say you break the process or the methodology early, as I have advised against that above. However, every posting comes with some sort of deliverable, and the sooner after you take your position that you can create absolute havoc with it, the more the developers and other team members will listen to you when you explain how to avoid that in the future. Or maybe they’ll just fear you’ll do it to them, so they’ll be more careful. Either way, making that splash early can really ramp up your influence.
   

• You don’t need a long contract or employment to make a lasting impression.
   
  

  Attila was sole King of the Huns for a mere eight years. In that short time the impact he made on his contemporaries was extraordinary. Much of this was due to his conquests. Eight years was a short period in which to hold both the Eastern and Western Empires to ransom with the threat of capturing and destroying both Constantinople and Rome, in addition to overrunning much of Germany and France.

  If you’re audacious and capable enough, you can make a lasting impression on an organization, its product, and its quality that will last long after you’re sacked for drinking kumis on the job on your third day. At the very least, they’ll remember the fermented mare’s milk.

I don’t care who you are or where you’re from, but if you’re from Wisconsin, you idolize Vince Lombardi, or you’re a heretic. He coached the Green Bay Packers, the small-market blue collar National Football League (fútbol norteamericano, not soccer, you international readers) and led them to something like 14 annual championships in 8 years. He was that good.

Now, you might ask yourself what a football coach from fifty years ago can teach you about quality assurance management. If you don’t read anything but quality assurance books, you’re only going to read what other people think. Crikey, you do understand you have to get outside those narrow channels of thought designed by someone else to synthesize your own knowledge, and reading outside the industry can do that. What are you, a Bears fan? (Patriots fan: If you weren’t in that lesser American conference, you wouldn’t be much better.)

At any rate, a couple lessons distilled from The Coach:

• The game is more complicated than it looks. You might think football is a couple seconds between whistles where big men hit each other. Below that tip lies an iceberg of individual team assignments working together in harmony and a whole glacier’s worth of individual and team preparation to handle any one game at a time. Like any software project or iteration, you need to remember the complexities and to account for them. A client wants a feature or a Web site? Sure, it sounds simple, and to the people who watch–that is, the clients or stakeholders–it is going to look short and easy. But if you’re managing it, you must not see it that way.
   
• Your team has different positions made of different individuals who are motivated differently. Sure, you’ve got a project manager/scrum master, and he knows his role and what he’s supposed to do. The team’s success and performing as well as possible so not to get cut are definite motivators. Be that as it may, you’ll get the best performance out of that player if you know the player as well as the role and know how much to micromanage, how much to lay off, and whether to shout or to whisper. To be a leader, you not only need to lead, but you need to follow the player–to learn each his needs and personalities.
   
• You have a chance to plan for each game. All projects are similar, following a set of constraints, but they all differ in the challenges faced within each one. Sometimes, the linebackers are fast. Sometimes the inside rushers can run right over your center. But the more you know about each game and each team, the better you can tailor your approach to the game or project. In college dorms, your developers could pick teams and start a pick-up game, but in the big leagues, it takes more than that.
   
  
• You can plan, but you must adapt to the game at hand. Your planning will have gaps. You planned all week to contain that #56, but he goes down with injury, and instead of a speedy safety, you’ve got a mountain of a man who knows his zones. The rain on gameday makes for a slippery ball. Circumstances arise that you might not have planned for, or maybe it’s just that you have to use a backup plan instead of the first plan. Requirements, timelines, personalities, and every element within your project are variables, not constants. They will change in the middle, and you have to adapt. Hopefully, your understanding of your team and the project at hand will give you a good head start on changing appropriately.
   
• You can adapt, but your success will come down to individual performance. You’ve got a plan, you can adapt, but if your developers turn in buggy code, you’re still sunk. You need to motivate the individual performers (see above) to get the best from them, and if they’re not working with the team, you still need to take action with that mouthy wide receiver.
   
• At the end of every project, you can find something to improve with the team. You beat the Bears 49-0? Great! But look how the left tackle missed a block on the sweep. You got the development in under budget and passed the application off to the testers only three days late? You can do better. Even when everything goes right, it could be made to go righter. In the next game or project, the improvement you put in place can be the margin of success if something else goes wrong.
   
• A hamburger and coffee is what a successful leader eats for lunch. Not sushi. That’s just pretentious.

That’s some of the reminders I got from this book, wrapped in a different metaphor (football) that makes it fresh. Your insights might vary. But, jeez, read something besides the industry standard material to make sure you think differently from the industry standard.

I recently read the book Genghis Khan and the Making of the Modern World to glean its insights into management and quality assurance in particular. As you would expect, I found a lot of good advice from the life of a man who drove natives into moats to fill them. Here are three:

• Prepare the environment.
  When Mongol reconnaissance groups were looking around at the edge of the largest contiguous empire the world has ever known, they would find places they were going to invade next year. Then they would burn the agriculture nearby and they would raze the villages in the manner of Genghis Khan. That way, next year or the year after, when the main of the Mongol forces would come riding through, the formerly developed land would be good pastures for the horses.

  This is not a call for adequate planning at the project level. The best plans evaporate in the heart of testing. It’s about looking ahead and making sure you have the resources to feed your horses when the time comes. Make sure you have the appropriate browsers installed and your VMs in a row before the projects start. Get a good basic idea in place for how you approach generic applications trampled into your people’s minds so that when they cross the river, they know what to do and where the forage is.

• Bring to bear different technologies and knowledge bases to devastating effect.
  The Mongols ruled a vast swath of Asia, the Middle East, and eastern Europe. They had a great diverse number of peoples upon whose knowledge they drew to great effect, using war machines and techniques learned in Asia to attack cities in Russia with something they’d never seen before. The Mongols also used scholars from all across the world to administer and maintain the vast trade lanes they created. If Genghis Khan had stuck to what he knew, fighting on horseback and herding, the Mongols would have remained a localized group of tribes doing what they always did.

  I think job ads that look for industry- or tool-centric knowledge automatically limit the teams that hire people who’ve walked the same happy paths (and unhappy paths) the existing team has always walked. Personally, I like a diverse team drawn from software covering many different industries for the insights into how software often fails in those other systems that might apply to the software in your industry that your company produces.

• When load testing, look for the indirect approaches to add load.
  Before Genghis Khan would attack a walled city, he would burn nearby hamlets and villages and farms and would drive the refugees before him to seek refuge in those cities. The influx of hungry and thirsty refugees would deplete the stored food and the cisterns. Their presence would tax the patience of the city dwellers and the powers-that-were who tried to keep order.

  By an indirect assault of adding a burden, Genghis found and exploited weakness.

  When you’re planning your load tests and performance assaults, don’t just focus on the normal pathways into the application. No doubt those gates and drawbridges are well defended. Don’t forget to look for other mechanisms that indirectly impair the performance of your application. A Web service or API that only one client will use (for now). A component that only contributes infrequently, but importantly. Don’t forget to make those small parts of your application wail, too, to see if their problems can cause big problems for your application.

I’m definitely heeding my own advice in going to history books for management lessons in QA, but all the material you learn and all you take in becomes grist for the creative mill. Or the destructive mill. Whichever sinks your boat.

When I’m not reading the Riot Act, I like a good novel or three. Here are some of my favorites dealing with the corporate world:

	Then We Came to the End: A Novel This novel centers on an ad agency in the downturn immediately after 2000 and how the people within interact with one another and their own fears as they wait for the axe to fall.
	Lloyd: What Happened: A Novel of Business An executive without a project becomes the intermediary between his company and the one about to buy it. As he reevaluates his life and his relationships, he has to fend off the temptations associated with corporate power. Or yield to them.
	You Look Nice Today: A Novel An executive heading up a corporation’s Total Quality program is sued by his secretary for sexual harassment based on innocuous comments and her emerging madness.

All three are satirical in tone and read much like prose Dilbert, but with depth.

[Spoiler ahead.]

Well, there it is. A novel about QA. Well, no, strike that, it’s a literary novel set in software development in the 1980s where a software defect is the MacGuffin and one of the main characters is a software tester.

Not by choice, oh, no. You can tell you’re reading a LITERARY book because the main character is clear to state that she has a doctorate in linguistics and only ends up working as a software tester when her academic career peters out. This is how you direct a book to an audience that does not work in the field the book covers, by saying ATTENTION! This character is the fish out of water and will explain to you the things you need to know because this character is just like you (in the field of academics or whoever reads Literary novels) except they work somewhere exotic (software development in the 1980s).

The book, as I mentioned, takes place at a database developer in 1984. They’re building a database from the ground up, including a GUI front end written from scratch that uses a mouse and everything. The main character, aforementioned, finds a defect that crashes the system, but is hard to replicate and appears to happen randomly, but usually in big demos. The book bounces back and forth between a first person double-effect narrator (since the story is told from present day ca. 2000ish after the dot.com bubble bursts, spurred when the narrator sees the one of the old screens still in use) and a limited omniscient narrator peering into the point of view of the developer to whom the defect is assigned.

The defect itself is a MacGuffin since the book deals mostly with the break-up of the tester’s and the developer’s personal relationship and the breakdown of the developer as he doesn’t think he’s a good developer and struggles to find this bug. That’s pretty much it. Enough technospeak and actual code and UNIX commands thrown into the mix, kind of like you’d find Hindi interspersed through a Kipling work to add authenticity. The tester finds self-actualization or empowerment by learning to code. The developer abruptly hangs himself, and even that did not cheer me. Uh, retroactive spoiler alert.

I didn’t care for it much, either for the plot nor the technical aptitude. The main characters are pathetic; I didn’t like either of them. The secondary characters are very, very stock and cardboard. The other testers are, essentially, the boss who is tough and fair minded and the other one whose dialogue mostly consists of her character tic of saying, “Meep.” The developers are mostly interchangeable except for their specialties, told to us of course. And I don’t know what the point of the frame story is, frankly. To show us the pathetic main character becomes rich but remains pathetic? Or to allow someone to feasibly set a book in 1984 when it’s written in 2002?

The best QA fiction is definitely genre fiction, mainly a thriller. Unfortunately, it hasn’t been written.

An ASQ mailing I just received:

Click for full size

 It’s not a children’s book preparing a child for a life of quality assurance.  It’s a children’s book about learning.

You know what makes a good children’s book for preparing children for a life in QA?  Grimm’s Fairy Tales, particularly the one where the children really do get eaten at the end of the story.  Or a good primer on Norse mythology, where the good guys lose in the end anyway.

What do you think would make a good children’s book for QAlings?

This is the third book in the trilogy (How to Break Software, How to Break Software Security, and this book).  This book, as you could guess by its title, focuses on applications written for the World Wide Web.

As with previous books, this one uses an enumerated list of “attacks” you can perform on the software you’re testing.  However, the “attacks” motif is a little misapplied, as the last chapters of the book are broadly constructed, including a chapter on “Privacy” and another broad overview of Web Services.

However, by focusing on differences between Web applications and regular applications (the stateless nature of the Web as well as exposed underlying technologies in the Web server and database server) that can help you transition if you’re used to working on desktop applications to the Web.  Heck, even if you’re already testing Web applications, you might find something in here to add to your repertoire.

However, read the book and apply the “attacks” in a triage fashion.  Sure, your pages’ source code might reveal something, but brothers and sisters, reviewing page code is the first attack (“Panning for Gold”) presented in the book.  It is not the first thing in my test plans.  As a matter of fact, given the projects I’ve worked on, reviewing source code has been spotty at best and falls in the test plan somewhere beyond spraying the Web server with a hose.  But if you have timeline enough and time, it is a good idea.

I bought these books for my team, so can expect I’d recommend them to you.  Go forth, buy them, and learn from them.

After I read How to Break Software (which a quick Google check indicates I have not reviewed, gentle reader, but most of you wouldn’t have read it anyway), I bought the companion volumes. This book, which I bought off of Amazon.com at its retail price, disappointed me where How to Break Software did not.

Both books run off of a quick list of fault-model testing (a term I learned from the first book). I had a ball with the first book, laughing at seeing some of my favorite dirty tricks encapsulated in someone definitive’s book. This book, however, didn’t hold the same glee for me.

The first book dealt with a broad subject and offered some very concrete things to try to attack software. This second book deals with a similarly broad subject (security testing), but is more abstract. The attacks it discusses aren’t as narrow and easy to recreate; they’re more methods and abstract ideas to try rather than concrete shortcuts to finding issues. I know, there’s something to be said for a broad, ranging methodology, but the first book wasn’t that way, and I didn’t expect this one to be that way. Additionally, the book is sized similarly to the first, which doesn’t allow it to go into a lot of detail for each of the abstract things it talks about.

Finally, I don’t know that the book focuses enough on actual security attacks; rather, it focuses on attacks that could be construed as security breaches. However, in many cases, they’re not specifically security attacks, but rather regular tests that could, if applied to applications needing security, be security attacks.

Maybe that’s all security testing is, but this book wasn’t different enough from the first book to make me wonder if it wasn’t really a sequel given a better title.

On the other hand, it does come with a CD and a tool which looks to be pretty cool, if I could get some professional time to play with it.

So buy the first book, How to Break Software, and apply its attacks to secure software. Buy this book if you’re really into it or if the company is buying it for you.