Archive for January, 2011
Matt Heusser (hoyser, not hoser, you hosers) writes a post about a feeling a lot of us have experienced: the sudden exhilaration when finding a bug you couldn’t have found following simple test cases, but that you did find:
It all starts with a feature called the profile. Now a profile in Socialtext is very similar to a profile in something like Facebook — it shows everything about one person in the system. Their name, contact information, tags, and what they’ve done lately. Here’s an example for our demo server:
A user’s profile in Socialtext
Notice that the profile events are all about the person. Seems pretty obvious, right? “Showing All Events from Me within (account)”
So this morning I was reading our internal blogs, mostly about the company face-to-face meeting last week. I read a comment by one person, who I’ll call Sarah. Sarah impressed me at the face-to-face, so I opened a new tab to tag her. I did, and someonething felt … wrong. I kept seeing my picture in her profile stream. This made sense, because I was the one tagging /her/, I had left a comment on her blog … but still …
I went ahead and made my own blog post, then went back to the tab. Sure enough, my blog post was now on her profile too — even though it had nothing to do with her.
On a hunch, I went into a test environment, created two browsers, logged in as userA, looking at my own profile, userB, editing pages. I waited three minutes and — BLAMMO — userB’s edits were showing up in userA’s profile.
It seems that the profile is restricted to the user you are viewing, but new events are not restricted.
Overall, this is not a terrible bug; unless you have thousands of users in your account, it is unlikely you’ll hang out on a profile page long enough to trip it. I mostly tripped it because I had the page open for an extended period of time in a different tab.
Would your test strategy find this?
You can have a good test strategy, or you can be lucky. That’s a false dilemma: you can be both. And a good tester seemingly makes his own luck.
While your test strategies might overlook some problems, you can increase your personal “luck” quotient and spot some problems and defects your test strategy would miss with good tester behavior:
- Be observant. Don’t just look at the thing you’re trying to confirm in each step of your test case. Notice the other elements of the screen and make sure they’re all doing what they’re supposed to. Make sure the tree view or folder view stays synched with the details pane. Make sure the cursor turns back to the pointer when it should. Make sure the right things are enabled at the right time. Even if the confirmation step of the test case just says The value displays in the edit box as masked.
- Be willing to improvise. When you’re running through a test, such as adding a user, if you see something wonky or suspicious, take time to go off script and investigate it. Remember, the applications are built to work right when you’re using it in the way prescribed by the requirements. It’s when you go off those well-worn paths that you find the wild things.
- Think outside the should happen. There’s a lot of understanding and assumption of what should happen if you follow the steps in the right order; that’s what most smoke tests like to prove. There’s a whole set of test cases out there that look to identify what does happen if you do something in an unexpected way. If you try to perform the steps in the wizard out of order, or if you go back and forth within a relatively streamlined process. When your application is all yin and you’re all bang.
- Understand the medium. If you’re on the Internet, your application is subject to certain limitations of the technology and needs to accommodate them. It needs to handle the fact that the user can hit the
stateless nature of the transactions. And that your authorization might be lost at any step of the process when you’re going through some sort of operation, or that a user might request any resource without authorization.
- Dibble in the software all the time. You might not be eating your own dog food (or what the dog next to you regurgitated, which is often a more apt metaphor), but you should be once in a while going off book into the application even if it’s not your job to test that feature that day. Just noodle a bit. Just look at it. Your perceptions of it will be leavened by what you’ve been doing recently, and you might see something differently. And if you hadn’t looked, you would not have seen it.
Oh, and one trick that I always, always do when the system has more than one user: See what happens when one user acts on a record while the other one is also acting on the record. The system better recognize and account for this possibility. Or that a user might be in the process of performing some action, and a supervisor might remove the user’s access in the middle of the user’s process.
But, Director, when would that happen?
Only when the user was fired and was in the system thinking about doing something very, very bad.
As I mentioned, you can make some of your own luck. And no matter how hep your testing strategies are, you’re going to need it.
I got this new Renoir calendar for the office (what, you expected I would only like the artwork of Derek Riggs?), and I’ve noticed that it has escaped the usual bourgeois constraints of comprehensible design. Behold:
You see, they “solve” the “real estate” problem of having a month comprise parts of six discrete weeks not by putting the final couple days on the bottom row with the preceding week, but by wrapping them to the top. You know, where you would not look for them because that’s the past. But the solution solves their problem.
When your designers break the mold and try to do something different, something that’s never been done before because now they have THE TECHNOLOGY to do it differently or merely because they’re just interns fresh from the second decade of the 21st century’s college of design pablum in their heads, you have to consider: Will my users understand this? Does it make their experience better? Or is it just something my designers want to do.
That leads me into some of my pet peeves:
- Placing the control labels into the controls. Especially irksome when I tab into the control and the label disappears.
- Reversing the normal button order of alert or confirm boxes. I hate when it’s Cancel/OK or No/Yes. Yes, I understand Macintosh has always done it that way, but it’s not the way I understand it. If I’m only half paying attention, which I usually am because I’m paying more attention to the all-woman Iron Maiden tribute band playing in the background, I’m going to do it wrong. Your application should be helping to keep me from messing up.
- Fields that are out of common order. You’ve seen forms where the Confirm Password edit box is not right after the password edit box or where the fields in an address are out of order. Okay, these are not so much design decisions, but sloppy oversights caught with no QA.
Bottom line: If you’re going to break out of the traditional metaphors your user has experienced for years or decades, you’d better make sure your application is doing it for your user’s sake, not for the bragging rights your designers will get for doing something outrageous or extreme.
If you want a chance to win a $50,000 ve-HICK-le (me, I want a new diesel quad cab full size pick-up truck), you can enter the Accelerate Into The Fast Lane Sweepstakes. You can enter on the Web, and you can see the official rules here.
The thoughtful designers here have used the common spoiler-hiding mechanism of making some text the same color as the background:
What’s hidden? The predetermined winner? The outside contest administrator’s home phone number? No! It’s the URL of the site you’re visiting?
Why did this happen? I can’t tell you. But I can tell you why it made it to the Internet that way:
Nobody looked at it.
You know and I know that this song describes the personal journey along the timeline, where development says they’ll have it for you on the 24th, but they meant that in hexidecimal, which means they’ll have it available for testing sometime after the go-live date.
This song is The River of Deceit by Mad Season.
The only direction we flow is down.
While paying taxes, I encountered this message, which apparently was written by a project manager:
The word process appears four times in roughly sixty words about the importance of not stopping the all-important process. Procedure makes another appearance. This totals 6-8% of the actual words.
Oh, and part of the picture is missing. How important or unimportant that piece is we’ll never know.
Sounds like project management to me.
Add the following date to your calendars and to your test cases: January 20, 2038:
The year 2038 problem (also known as Unix Millennium Bug, Y2K38, Y2.038K or S2G by analogy to the Y2K problem) may cause some computer software to fail before, in the year 2038 or after. The problem affects all software and systems that both store system time as a signed 32-bit integer, and interpret this number as the number of seconds since 00:00:00 UTC on Thursday, 1 January 1970. The furthest time that can be represented this way is 03:14:07 UTC on Tuesday, 19 January 2038. Times beyond this moment will “wrap around” and be stored internally as a negative number, which these systems will interpret as a date in 1901 rather than 2038. This is caused by Integer overflow.
In the end, all software shortcuts will out and will crash a moonplane.
I know, a lot of companies like to build their sites around CMSes and then turn the actual content entry into an intern. Once that CMS is humming along, they think they’re golden.
Until it looks like an incomplete Mad Lib:
Jeez, Louise, if that process is anything like what you’re doing, if you’re not going to have someone in your QA staff proofreading, at least make sure another intern looks over the content before it’s published.
Just because someone owns a computer that could handle the moonshot in 1969 doesn’t mean he or she owns a computer that can handle the extent of your cutting edge design genius:
At the RISK of REPEATING MYSELF to an entirely DEAF DUMB AND STUPID — THOUGH APPARENTLY NOT BLIND BECAUSE THEY HAVE TO HAVE THEIR FRICKING VIDEOS DON’T THEY — internet, I will say this again:
NOT EVERYONE ON THE PLANET HAS A TEN-THOUSAND DOLLAR COMPUTER WITH 75 TERABYTES OF RAM AND A HARD DRIVE CAPABLE OF PILOTING THE SPACE SHUTTLE NOR DO THEY HAVE A SCREAMING FAST INTERNET CONNECTION CAPABLE OF TRANSMITTING PURE THOUGHT TO OTHER GALAXIES IN UNDER A NANOSECOND. SO FUCKING TRIM YOUR WEBSITES, OR IF YOU ARE THE “WATCH THIS KEWL VIDEO HURR HURR” LINKERS, WARN OTHERS OF THOSE FUCKERS SO WE WON’T BOTHER TO CLICK AND HAVE OUR BROWSERS CRASH AND SEE THAT STUPID “YADDAWARE CRASHED, WOULD YOU LIKE TO NOTIFY MICROSOFT” THING ON THEIR DESKTOP.
If you’re building 3-d Flash immersive environments to sell cat food, more often than not, you’re trying too hard. Don’t only think of what all you can do with your high-end Macintoshes there in the office. Remember, Roberta is out there, limping along on a low end Windows XP box and trying to get a coupon to save her fifty cents on her next purchase of her product. If she can get it.
An IBM ad with the copy “What 3 million lines of code means to a piece of luggage”:
Apparently, somewhere along line 2,999,985, it means your luggage gets sucked into a jet engine, turned into confetti, and sprinkled onto Amsterdam like what Joe Strazzere would call “jimmies.”
There’s a tagline for QA: QA: If We Can Imagine It, Some User Will Accidentally Do It.
In the straight talking that is QA, I’m reminded of the following commercial:
It’s not a collection of proactive technologies designed to engage target audiences to facilitate efficient markets. It’s a Web site. Call it that.
I know how Taio Cruz feels when he sings Dynamite.
I know, I know, it’s lightweightish for QA music, but I’ve heard it both at the gym and the dojo. So some of the frails dropping 125lb dumbbells and doing sweeping kicks think it’s worth something. Also, it mentions explosives right in the title.
Yesterday, I picked on a sort order in a country drop-down list. But there’s another thing to consider external to the list: Is your list of countries correct?
For example, if your country codes list says Netherlands Antilles like this:
You’re now out of date:
The acts of parliament integrating the BES islands (Bonaire, Sint Eustatius and Saba) into the Netherlands were given royal assent on 17 May 2010. After ratification by the Netherlands (6 July), the Netherlands Antilles (20 August), and Aruba (4 September), the Kingdom act amending the Charter for the Kingdom of the Netherlands with regard to the dissolution of the Netherlands Antilles was signed off by the three countries in the closing Round Table Conference on 9 September 2010 in The Hague.
You need to add Curaçao (not to be confused with the Japanese director Kirosawa, although it might improve tourism), Bonaire, Saint Eustatius and Saba, and Sint Maarten (Dutch Part), which is the Netherlands’ half of St. Martin. Leaving St. Martin with one entry is akin to putting Hispaniola or Haiti in the list and expecting the people from the Dominican Republic to just answer that.
You can find the latest ISO Country list here to see how you’re doing.
And you can already start writing your defect for the lack of an entry for South Sudan right now and leave it open in your browser window to submit soon.
When I as nominating myself as a great tester and hoping to win a free conference, I saw the sort order on the Country list:
Poor Sweden. What did they do to offend Software Test Professionals to be dead last?
Remember, take a look at your Country drop-down lists and make sure they’re sorted right and that they’re spelled right. Look at the VIRGIN ISLANDs BRITISH, for crying out loud.
Here’s an online QA aptitude test.
I got 72%.
Of course, I gamed the controls a little. So who knows how apt I am.
Are you a meek, quiet QA analyst, sitting in the meetings and only listening to the developers thump their chests or, more likely, stroke their Van Dykes as they make excuses?
THEN YOU’RE NOT QA!
QA is Loud ‘n’ Proud, like Pretty Maids!
Albeit most of the time we’re less exclamation pointy. We leave that to marketing.
Other testers recount their experiences with finding themselves in test data found in production:
- James Bach has a vanity license plate with TESTER on it and got a ticket as a result. Word is that this used to happen a lot with plates with NO PLATE or NONE on them, too, since the cops wrote this on tickets given to those cars and HQ added an extra fine for not having a plate.
- As an overcorrection, Eusebiu Blindu (aka testalways) found that his hosting provider blocks the keyword test in certain circumstances.
As you can well guess, I insist upon testing in production whenever possible, and when I was working for an interactive agency, I entered the contests as Brian Tester with an e-mail address with the interactive agency’s domain and with the agency’s address. As a result, I, I mean Brian Tester won the consolation prizes in several of the CPG client’s giveaways. This meant coupon packs and the occasional free sample. The biggest prize I ever got was a shower caddy, a piece of plastic you could doublestick-tape to your shower to hold the client’s product bottles and useless for anything else.
These amusing anecdotes aside, what can one do to try to prevent this?
- If possible, remove your test data. If you can delete your test record as part of the test, do it. In many cases, though, you don’t have a delete function or direct access to the database to delete.
- Standardize your test data for production. Use the same surname or address whenever you test, for example. This provides a key to use to identify and scrub that test data whenever your organization needs to do that test data. You could use your company’s address, for example. If you can’t find a single field you can be sure is 100% unique and that users will not actually input, use a composite built of several fields.
- Understand where the data goes. A lot of times, other companies and organizations are getting dumps of your application’s data for other purposes, such as contest administration. Even if your team knows to scrub for your data, these other organizations might not. And suddenly, you’re on the top 10 most wanted list, Bob Tester. Sorry.
- Communicate with data consumers. Once you know where the data is going, you can tell them about the standard form for your test data and warn them to scrub it or handle it.
- Document your test data. As part of any data schema or spec document that should get passed onto other data consumers, especially the ones about whom you won’t know or with whom you will not communicate, include details about your test data standard. Include it in code comments. Stick that note everywhere as a warning.
Following those steps, if you can, might help prevent the problems that happen when some program takes your test data seriously.
Today’s word for use in tomorrow’s status meeting: Sooterkin.
A sooterkin is a fabled small creature about the size of a mouse that certain women were believed to have been capable of giving birth to. The origin of this initially jocular fantasy lies in the 18th century, and came to be considered factual by some eminent physicians of the day. It is attributed to a tendency of Dutch women to use stoves under their petticoats to keep warm, hence causing the breeding of these small animals, which when mature would be born.
A sooterkin may also be used to refer to an abortion, an abortive scheme, or a failed plan.
(Word seen here, although not in context.)
It’s just a message one sees very briefly when configuring, so that means your team might see it once during the test phase? Is that a reason or an excuse?
Given that it’s one of the first messages that your user sees, this will be one of the first impressions the user gets as to the reliability of your software. And if your messages are slop, your user–or potentially former soon-to-be user–might think your software is slop, too.
Your software’s installer is an application just like your application is. Test it vigorously.