QA Hates You

Maybe he didn’t coin it, but Joe Strazzere talks about how QA needs to do some Crappy Path testing.

I draw your attention to this post from January 2009 about another type of test case to consider during leap year.

Not only do you have to accommodate the date of February 29, 2012, but you need to also check any calculations that count the days.

There is a Unicode character and an HTML character for the skull and crossbones.

Please work it into your testing accordingly.

The real world intercedes with something that would never happen in the real world:

THERE is no today in Samoa.

The tiny nation will jump forward in time as it crossed westward over the international dateline to align itself with its main trading partners throughout the region.

At the stroke of midnight on December 29, the time in Samoa will leap forward to December 31 – New Year’s Eve. For Samoa’s 186,000 citizens, Friday, December 30, 2011, will simply cease to exist.

I wonder how many automated processes melted down. Or are still going to melt down.

Remember to test all of your future applications that allow you to select a birthdate and country or a start/end date and country that this particular rule should exist.

Oh, man oh man, I can’t wait to log my first defect and start my first fight over it.

(Courtesy Trisherino.)

A pretty stock naughty thing to do when testing a Web application is to double-click a link instead of single-clicking it.

But, Director, what sort of madman would do such a thing?

• Someone used to the desktop paradigm might do it just because he or she does not know not to (someone like Roberta).
• Someone like me who doesn’t see any action immediately and wonders if he clicked the link or if he clicked while the cursor was not on the link.

Case in point: In WordPress, you can move an item to the trash by clicking the link labeled, appropriately, Trash:

If you click the link, the page comes back with the item missing from the list and your trash incremented by 1.

If you double-click the link, though:

Hilerrority ensues! The application deletes it and then tries to delete it again! This results in an unspecific error condition, but what would happen in your application?

Come on, guys, the user might double-click a link, and your Web application needs to take that into account and to handle it elegantly. More elegantly than a non-specific error message with no further navigation, certainly.

Add the following date to your calendars and to your test cases: January 20, 2038:

The year 2038 problem (also known as Unix Millennium Bug, Y2K38, Y2.038K or S2G by analogy to the Y2K problem) may cause some computer software to fail before, in the year 2038 or after. The problem affects all software and systems that both store system time as a signed 32-bit integer, and interpret this number as the number of seconds since 00:00:00 UTC on Thursday, 1 January 1970. The furthest time that can be represented this way is 03:14:07 UTC on Tuesday, 19 January 2038. Times beyond this moment will “wrap around” and be stored internally as a negative number, which these systems will interpret as a date in 1901 rather than 2038. This is caused by Integer overflow.

In the end, all software shortcuts will out and will crash a moonplane.

An old Blockbuster envelope teaches us a valuable lesson about alternative methods of output:

Click for full size

So what portions of your application come out of the printer? Does it work right? Does it look right? Is it correct?

It’s not enough that you make sure the print dialog comes up correctly. You need to make sure that the extras that are often added to the printed page display correctly. For example, some maps add details such as the location, some Web sites put their names on it, and some applications use formula. To ill effect in this case.

If you want to be a real rapscallion, see what happens if you print to a file or to a PDF driver of some sort. Because someone out there in the real world just might.

Friends, we’ve already covered file upload test cases, haven’t we?

Well, if you’re new here, let’s recap:

• Large files: make sure the application can handle 1Gb or more or stops user from uploading them.
• Empty files: make sure application can handle 0 kb files.
• Invalid files: make sure application can handle files that are corrupted.
• Wrong file types: make sure application can handle when you try to upload the wrong type of file
• Long file names: make sure application can handle long file/path names.
• Invalid file/path: make sure application can handle invalid locations if you’re allowed to type in file names and paths.

The little Space Your Face Flash ditty created by NASA hangs on the 2nd and 3rd of the bullets above:

Click for full size

Worse, it hangs with some cheaply produced space groove playing.

So your designers have constrained the input length on your application so you cannot enter more characters than the database can handle. If the developers force the string into all caps, have I got a nasty little trick for you. Ladies and gentlemen, the German eszett:

Also, the eszett or scharfes S (ß) is used. It exists only in a lowercase version since it can never occur at the beginning of a word (there are a few loan words starting with an s followed by a z (e.g. Szegediner Krautfleisch but that is not the same as the eszett which counts as one letter).

In all caps it is converted to SS….

There’s a new unicode symbol for the capital version, but a lot of old applications will still force that into an SS. So a word like confuße might get uppercased to CONFUSSE, and if you set the string to the maxlength, uppercasing it will blow that up.

To be honest, I did discover this when I was working on an application for a German customer and I (and only I of a team of far more seasoned QA people than I at the time) sought out the German alphabet to learn its vagaries.

I just ruined a little of my mystique, didn’t I?

However, if your application might possibly be localized to German, you have my permission to use this. Use this new power only for good. Strangely, though, QA good means evil to everyone else, but that’s not our fault.

Here’s a SQL Injection Cheat Sheet for you.

Remember to check your form fields for these bad dogs when you can.

(Link courtesy the Twitterverse.)

Based on a tweet this morning lamenting the dearth of proper test plan sample documents on the Internet, I put together a sample document in PDF format that you can use when putting together your own test plans.

You can view that sample here.

I hope that my regular readers and especially those of you who got here by a Google search find it useful for your testing documentation.

So I stopped by the Branson (Missouri) Regional Airport recently, and I spotted this kiosk:

Click for full size

It offers the user the opportunity to enter some sort of contest to go to Nashville. It’s obviously a Web browser in kiosk mode, but this one has a full keyboard with a trackball and two mouse buttons. Uh oh.

So I click the Contest Rules link at the bottom and get the contest rules, which has a naked link at the top that takes you back to the form. But hover over the link and right click and…. Uh oh.

Click for full size

What happens if I open that in a new window? Hello, Internet!

Click for full size

So a user has complete access to the Internet. Go where you want. Get all the malware you want. I didn’t try to see if a regular download and install worked, but I would not doubt it. What happen if I ALT+TAB?

Click for full size

Lookie there! Lookie there! It’s the command line. A little CTRL+C action and I have access to issue commands to the machine and maybe even the network.

So is that Cat-5 cable running out of the back of the box connected to the airport network itself or a dedicated safe portal to the Internet? Given what we’ve seen here, what do you think?

If you’re ever called to check out a kiosk application, not only should you run through the form the kiosk will host, but you should get a kiosk itself and run it through its paces and look outside the confines of the application to look for security pitfalls.

You need to check out the user interface action. This kiosk gives the user all the normal tools that users need for full input opportunity to the Internet. Some kiosks only have touchpads or touchscreens. Here are a couple of things to think about:

• Know your keyboard shortcuts. Most people don’t know these keyboard shortcuts, but they do things to your active window (even your kiosked browser). What can you do with that?
• Know your internal browser behavior. I remember seeing a kiosk with only a touchscreen that offered the Web sites of a building’s residents. Within a touchscreen environment, you would think you’re limited to navigating through links in the browser window. You would be wrong. mailto: links trigger the helper application associated with e-mail. What can you do when you try that?
• What happens when you unplug the machine and plug it back in? It reboots, probably, affording you the ability to go into alternate bootup scenarios and whatnot. Should your user have access to that? Probably not.

To begin vetting kiosks, you need to think outside the terms of your application and think in terms of the technologies that encapsulate it. The better you understand those and can identify the ways users could interact with the whole kiosk, the better you can prevent them from doing so inappropriately.

If you’re anything like me, you use e-mail addresses for testing purposes. I make up nonexistent addresses for user creation and use one or more existing e-mail addresses that I receive in my inbox for tests where I need to review the resulting e-mail, such as a tell-a-friend e-mail or a form that elicits an automated response such as a customer service ticket.

But what happens if you put in the return e-mail address of your company’s newsletter?

In certain circumstances, when your organization composes and compiles those e-mails on its own, you might find that entering the newsletter return address in one of your organization’s other automatic e-mail generating applications will trigger an e-mail to your entire newsletter list or some other e-mail, such as an open relay response.

It’s a damn dirty trick, and you should try it on your organization before someone else does.

As a rule, your organization should make sure that the user cannot enter those sorts of e-mail addresses, but it should allow you to test using individual e-mail addresses internally.

Here’s someone thinking like QA:

Sure, the keyboard, mouse, and data import features are some of the several obvious ways that data gets into your system. Are there any others? Scanners perhaps? Photo recognition or OCR?

You’ve got to be devious to be a tester whom I respect.

Thomas Construction offers a $75 gas cards to people on a direct mail list.  Users can visit a Web site to sign up for the program, and the URL for the site uses the name on the direct mailing as a subdomain instead of as a querystring parameter.

For example, B– here gets his information prepopulated:

Click for full size

Now, if you go to the www subdomain, you are recognized as a guest:

Click for full size

Now, you know what the first thing I would check and one thing that nobody else would check at Thomas’s interactive agency, don’t you?

(more…)

If you’re testing file uploading or attachment capabilities, don’t forget to try empty files and corrupt files to see if your application can handle them appropriately.

Here’s a handy tool called File Destructor that creates invalid files with different extensions of determined size that you can use when running your corrupt file tests.

It’s designed to create files you can send to teachers to support a “the computer ate my homework” excuse, but we in QA can subvert that, can’t we? We can subvert anything.

The developers of the Circuit City store locator fail logic.

Here’s the situation.  You’re a user in tiny Grantwood Village, a mostly forgotten municipality in St. Louis County, Missouri, who wants to go to Circuit City because….well, okay, maybe it is an outrageous use case, but it fails:

1. Go to the Circuit City home page.
2. Click the Store Locator link at top.
3. Store Locator displays:
   
   In the City edit box, type grantwood village.
4. From the State drop-down list, select Missouri.
5. Click Find.
6. Uh oh.  According to Circuit City, Grantwood Village does not exist:
   Much to the chagrin of Grantwood Village.Well, then, type the zip code of Lakeshire, Missouri (63121) into the Zip code edit box.  Funny, though, Lakeshire is even smaller than Grantwood Village, as it’s essentially a small subdivision with a post office.
7. Click Find again.
8. The application acts as though the zip code is invalid:
   

This occurs whether you click the Find button underneath the Zip code edit box or underneath the City/State combination.  Don’t get me started about the design wisdom of putting two controls on a form that do the same thing.  You cannot convince me of its utility, and I disbelieve in your value of symmetry.

In this form, if the application detects a value in the latter, it ignores the former, period.  So it does sort of handle Or (you need to enter something in one or the other), it does not handle both (And) correctly.  Even though someone will probably encounter the situation of entering data in both forms.

And, when you’re feeling particularly nasty (which is to say, every day of the week), remember to try 87894.  This is an invalid zip code, and if your application doesn’t handle nonexistent zip codes (not merely strings that are not five numbers) or relies on a Web service call or whatnot to an application that does not handle nonexistent zip codes, hilarity ensues.

Nested within a Daily WTF story, we find an interesting test condition.

“But what if you just, say, pull the plug? A Finally block won’t execute when the computer is turned off!”

If you need me, I’ll be in the server room.

When you’re testing an application with any sort of security, you test the following as a matter of course:

• User with correct username/password can log in.
• User with incorrect username/correct password cannot log in.
• User with correct username/incorrect password cannot log in.
• User can log out.
• User who is not logged in cannot access protected functions.

However, in the case of some applications and most Web applications, the server has a time limit on user inactivity; that is, after a certain amount of time, the server assumes that the user is done and shuts off the connection. You better make sure that works.

(more…)

As some of you know, the old calendaring system in use with certain Western countries from Roman times, called the Julian calendar, had some problems with not keeping up with the sun or something esoteric. To correct this, the Church made some adjustments to leap years and whatnot and blah blah blah (you want the details, go to Wikipedia).

However, this little bit of historical trivia lends itself to some fun with your date entry fields.

(more…)