It’s Better To Be Lucky Than Good. Fortunately, QA Is Both

Matt Heusser (hoyser, not hoser, you hosers) writes a post about a feeling a lot of us have experienced: the sudden exhilaration when finding a bug you couldn’t have found following simple test cases, but that you did find:

It all starts with a feature called the profile. Now a profile in Socialtext is very similar to a profile in something like Facebook — it shows everything about one person in the system. Their name, contact information, tags, and what they’ve done lately. Here’s an example for our demo server:

A user’s profile in Socialtext

Notice that the profile events are all about the person. Seems pretty obvious, right? “Showing All Events from Me within (account)”

So this morning I was reading our internal blogs, mostly about the company face-to-face meeting last week. I read a comment by one person, who I’ll call Sarah. Sarah impressed me at the face-to-face, so I opened a new tab to tag her. I did, and someonething felt … wrong. I kept seeing my picture in her profile stream. This made sense, because I was the one tagging /her/, I had left a comment on her blog … but still …

I went ahead and made my own blog post, then went back to the tab. Sure enough, my blog post was now on her profile too — even though it had nothing to do with her.

On a hunch, I went into a test environment, created two browsers, logged in as userA, looking at my own profile, userB, editing pages. I waited three minutes and — BLAMMO — userB’s edits were showing up in userA’s profile.

It seems that the profile is restricted to the user you are viewing, but new events are not restricted.

Overall, this is not a terrible bug; unless you have thousands of users in your account, it is unlikely you’ll hang out on a profile page long enough to trip it. I mostly tripped it because I had the page open for an extended period of time in a different tab.

Heusser asks:

Would your test strategy find this?

You can have a good test strategy, or you can be lucky. That’s a false dilemma: you can be both. And a good tester seemingly makes his own luck.

While your test strategies might overlook some problems, you can increase your personal “luck” quotient and spot some problems and defects your test strategy would miss with good tester behavior:

  • Be observant. Don’t just look at the thing you’re trying to confirm in each step of your test case. Notice the other elements of the screen and make sure they’re all doing what they’re supposed to. Make sure the tree view or folder view stays synched with the details pane. Make sure the cursor turns back to the pointer when it should. Make sure the right things are enabled at the right time. Even if the confirmation step of the test case just says The value displays in the edit box as masked.
  • Be willing to improvise. When you’re running through a test, such as adding a user, if you see something wonky or suspicious, take time to go off script and investigate it. Remember, the applications are built to work right when you’re using it in the way prescribed by the requirements. It’s when you go off those well-worn paths that you find the wild things.
  • Think outside the should happen. There’s a lot of understanding and assumption of what should happen if you follow the steps in the right order; that’s what most smoke tests like to prove. There’s a whole set of test cases out there that look to identify what does happen if you do something in an unexpected way. If you try to perform the steps in the wizard out of order, or if you go back and forth within a relatively streamlined process. When your application is all yin and you’re all bang.
  • Understand the medium. If you’re on the Internet, your application is subject to certain limitations of the technology and needs to accommodate them. It needs to handle the fact that the user can hit the stateless nature of the transactions. And that your authorization might be lost at any step of the process when you’re going through some sort of operation, or that a user might request any resource without authorization.
  • Dibble in the software all the time. You might not be eating your own dog food (or what the dog next to you regurgitated, which is often a more apt metaphor), but you should be once in a while going off book into the application even if it’s not your job to test that feature that day. Just noodle a bit. Just look at it. Your perceptions of it will be leavened by what you’ve been doing recently, and you might see something differently. And if you hadn’t looked, you would not have seen it.

Oh, and one trick that I always, always do when the system has more than one user: See what happens when one user acts on a record while the other one is also acting on the record. The system better recognize and account for this possibility. Or that a user might be in the process of performing some action, and a supervisor might remove the user’s access in the middle of the user’s process.

But, Director, when would that happen?

Only when the user was fired and was in the system thinking about doing something very, very bad.

As I mentioned, you can make some of your own luck. And no matter how hep your testing strategies are, you’re going to need it.

Comments are closed.

wordpress visitors