QA Hates You

So I installed the new full CD version of Intuit QuickBooks, which is adware designed to get you to buy a lot of Intuit additional services disguised as accounting software. Now, if you’re like me, you’re not into the intricacies of actual accounting nor the myriad business rules that the various state and Federal governments change upon a whim, but you rely on software and a good accountant (or, sometimes, an accountant, although I’d like to add my current accountant is a good accountant unlike previous engagements who continue to bill me a small amount every year for simply having my address in their files).

Where was I? Oh, yes. I was talking about trusting your application, particularly one with complicated rules whose violation might result in a prison sentence. You want to trust that application, don’t you? So do I.

But I get the software installed and get into the mandatory registration (that is, give us personal information so we can target more in-application advertising pop-ups to you), and I get confronted with obvious slops on the design.

To whit:

A couple missing lines and slurred text, probably caused by poor compression or sizing.

Next up:

A stray bracket in the corner.

Man, oh man, I can’t wait to find out what strange punctuation marks it leaves in my figures.

Do I trust the application? Not so much. Which is why I don’t use it for much more than a glorified check register. And if it continues with its unrepentant, unrelenting barrage of “Collect credit cards with Intuit!”, “Print checks with Intuit!”, “Let Intuit have access to all your financial accounts!” banners popping up before I can pay my bills, I won’t have to trust it in the future, as I move to Microsoft Excel where it’s nice and quiet.

You know, when this blog first started, I used to noodle around existing Web sites, find errors on their Web forms, and do some commentary on them. I haven’t been in that habit for a while, so you might think the quality of applications in the wild has improved. Oh, but no.

Take the Progressive Insurance contact us form:

Now, if you don’t select a topic before you click Submit, it asks for a topic. But only a topic

When you select a topic, it changes the fields on the form to reflect what they want for that kind of inquiry (enquiry for our R.P. friends). If you click Submit then, it shows you a list of the fields you need to fill out:

But if you click the Reset button to reset the form, well, that’s not cricket (cricket for our R.P. friends):

You know, any time your form changes the controls on the screen due to AJAX or other techniques, it’s a different form. And it never hurts to check your reset button in various forms of filling out the form, especially if there are look-ups or state changes as you fill out the form.

UPDATE: Welcome, Progressive Insurance readers!

You might know, if you’re in the United States, have health insurance, and have an insurer that uses Express Scripts for its prescription benefits management, that as of January 1, 2012, the Walgreens pharmacy chain and Express Scripts contract ended. Which means that your insurance card does not work at Walgreens, and if you want our pharmaceuticals at reduced rates, you have to transfer your prescriptions to other pharmacies.

How many prescriptions are affected?

90,000,000:

An Express Scripts spokesman say their customers previously filled 90 million prescriptions at Walgreens. Now they’re taking them elsewhere.

It’s my understanding that the Express Scripts processing system was down all day on January 11. Is it related? I don’t know.

But I do wonder whether Express Scripts load tested its system to handle 90,000,000 change orders in a matter of days while handling its normal processing for all other normal maintenance.

It could be a valuable lesson anyway: Even after you’ve load tested your application to the limits of your budget for virtual users or to a level where the stakeholders are comfortable with very gradual ramp up times, sometimes events out of IT’s control could lead to a catastrophic meltdown.

So I was playing Rampage World Tour on the PlayStation 2 because I am 15 years behind in my gaming.

After destroying 99% of the first city (what, I missed some part up to 1%?), I got an error:

What? There are other lockup errors one can find? I’ll have to try them to collect them all!

While I do that, join me in recasting the lyrics to this song to fit the occasion. Consider it your Friday software testing exercise:

A Cracked list: The 8 Creepiest Glitches Hidden In Popular Video Games.

An element of the recent high-profile American trial of Casey Anthony that should be of interest to us:

Assertions by the prosecution that Casey Anthony conducted extensive computer searches on the word “chloroform” were based on inaccurate data, a software designer who testified at the trial said Monday.

The designer, John Bradley, said Ms. Anthony had visited what the prosecution said was a crucial Web site only once, not 84 times, as prosecutors had asserted. He came to that conclusion after redesigning his software, and immediately alerted prosecutors and the police about the mistake, he said.

…

Concerned that the analysis using CacheBack could be wrong and that a woman’s life might be at stake, Mr. Bradley went back to the drawing board and redesigned a portion of his software to get a more accurate picture.

He found both reports were inaccurate (although NetAnalysis came up with the correct result), in part because it appears both types of software had failed to fully decode the entire file, due to its complexity. His more thorough analysis showed that the Web site sci-spot.com was visited only once — not 84 times.

Remember, those defects your developers mark as “Known Issue” or reject still can have real-world effects.

The Wall Street Journal has an article about cascading failures in airline computer systems:

Whether it’s caused by a power failure or a possum, an airline computer outage unleashes more problems more quickly than similar breakdowns in almost any other consumer business.

A recent spate of technology glitches at U.S. airlines has inconvenienced thousands of travelers, spawned long airport lines, delayed or canceled flights and led to a wave of negative publicity.

United Continental Holdings Inc.’s United Airlines unit suffered a meltdown June 17 that forced it to cancel 36 flights and delay another 100. US Airways Group Inc. was hit by separate glitches on June 10, 18 and 19.

Alaska Air Group Inc. had its turn back in March, canceling 150 flights affecting more than 12,000 customers. Earlier in March, Southwest Airlines Co. experienced two separate technical foul-ups within two days, although the one related to the rollout of its revamped frequent-flier program didn’t delay flights.

With packed planes leaving little room for error, airlines are trying myriad upgrades and other solutions to keep their computers running everything from flight dispatching to crew scheduling, passenger check-ins, airport-departure boards, ticket sales and frequent-flier programs.

Cascading failures caused by unforeseen, unrelated, or untested circumstances. Now you know why some of us use milk of Magnesia as our coffee cream.

Two sellers on Amazon dance to the beats of their own algorithm:

A few weeks ago a postdoc in my lab logged on to Amazon to buy the lab an extra copy of Peter Lawrence’s The Making of a Fly – a classic work in developmental biology that we – and most other Drosophila developmental biologists – consult regularly. The book, published in 1992, is out of print. But Amazon listed 17 copies for sale: 15 used from $35.54, and 2 new from $1,730,045.91 (+$3.99 shipping).

I sent a screen capture to the author – who was appropriate amused and intrigued. But I doubt even he would argue the book is worth THAT much.

At first I thought it was a joke – a graduate student with too much time on their hands. But there were TWO new copies for sale, each be offered for well over a million dollars. And the two sellers seemed not only legit, but fairly big time (over 8,000 and 125,000 ratings in the last year respectively). The prices looked random – suggesting they were set by a computer. But how did they get so out of whack?

When you’re building something that reacts to events in the cloud, you’re going to get some strange events. You’ll need to consider some alternative workflows, such as unreliable input and buggy software dependencies.

Because this example shows another “Never” happening.

(Link seen on Twitter.)

A receipt from a car wash that accepts credit cards shows a stunning amount of inaccurate data:

The name, address, and the approval number are all obviously dummy data. Should your system in production be outputting this? Of course not. But do you let the users–in this case, an installer or an administrator of the kiosk–just use the dummy data?

You see this trap sometimes when applications put the labels for controls as text in the controls themselves, such as an edit box that says “First Name” until you type into it. Sometimes, you’ll find the application will check to make sure the edit box is not empty, but the application is perfectly happy with “First Name” in it. The application is happy, but is the client happy that 50% of his registrations come from First Name Last Name of Address City State 55555? I think not. Don’t let them do it. Even if they’re trusted computer professionals.

Secondly, this is another reminder to check all your application’s outputs, QA. I know, that means sometimes getting up from the faintly warm glow of your monitor and the seat that has molded itself nicely to your backside, but if your application prints anything, you’d better make sure it looks good on paper (and on A4 paper if you’re pretending international use).

Hotmail has a mechanism by which it translates URLs found in its text e-mails into hyperlinks. It does this smartly, using a special logic that I make sing the algorhythm-and-blues:

Click for full size

You can forensically see what happened here. The logic says URLs don’t ever end with periods, so if there’s a period at the end of the URL, it’s the period at the end of a sentence and don’t make it hot.

Except, of course, in this case the period at the end of the sentence is part of the last parameter on the querystring, a user name that ends in a period. And if the period is not passed along when the user clicks, it’s an invalid URL.

On the plus side, it doesn’t look like a Hotmail bug; it looks like a problem with the receiving server. W00t!

You Microsofterati reading the blog ought to log that in your Hotmail buddies tracker for them.

It’s just a message one sees very briefly when configuring, so that means your team might see it once during the test phase? Is that a reason or an excuse?

Click for full size

Given that it’s one of the first messages that your user sees, this will be one of the first impressions the user gets as to the reliability of your software. And if your messages are slop, your user–or potentially former soon-to-be user–might think your software is slop, too.

Your software’s installer is an application just like your application is. Test it vigorously.

So I recently decommissioned one of the PCs here in the lab. As part of it, I went through the Adobe process to decertify Dreamweaver on one machine, uninstall it, reinstall it on the new machine, and recertificate it. Then I created a shortcut of it on the desktop, and I ran it:

Oops.

Well, I wasn’t going to spend an hour or so on the phone trying to fix it, so I uninstalled it and reinstalled it on the new machine, and when I tried to run it again, I got the same catastrophic error message.

Was it time to call customer support and face a bullet of whatever precious metal Adobe uses? Of course not! I’ll take another look and….

Instead of creating a shortcut on the desktop, I’d copied the executable there and tried to run it, and the poor little application was lost in the big forest without its dependencies.

Yeah, there I was, a computer professional making a mistake that rendered Adobe Dreamweaver speechless.

So, what does your application do when it cannot find its dependencies? If it’s anything like this particular application, it spits up an unrelated error message and it asks you to call support. And support will have a werewolf of a time trying to figure it out.

So play around with your executable locations, including putting dependencies on network drives and not connecting to them, altering paths, maybe renaming directories without changing them–particularly if you can change the name of a directory from within the application and the application stores the hard file path somewhere within it (I used a testing tool once that let me do that, and then it could not find its own tests).

Granted, you’re not going to convince many people that these are high priority defects, but it’s something you should still consider when trying to build a quality application.

An Indiana resident reports some crazy swings in the weather this year:

Another reminder that your monitor is not the only output you need to test for your application.

(Seen here.)

An old Blockbuster envelope teaches us a valuable lesson about alternative methods of output:

Click for full size

So what portions of your application come out of the printer? Does it work right? Does it look right? Is it correct?

It’s not enough that you make sure the print dialog comes up correctly. You need to make sure that the extras that are often added to the printed page display correctly. For example, some maps add details such as the location, some Web sites put their names on it, and some applications use formula. To ill effect in this case.

If you want to be a real rapscallion, see what happens if you print to a file or to a PDF driver of some sort. Because someone out there in the real world just might.

Since yesterday’s New Twitter bug proved so popular, let’s look at another one. You can recreate this one on your own using the following steps in Mozilla Firefox:

1. Click the Retweets link.
2. The Retweets menu displays. Click Your Tweets, Retweeted.
3. Your retweets, assuming you’re some fraction as clever as I am and someone else repeated what you said, display. Highlight one and notice the little more info subpanel caret that displays. I don’t know what Twitterians call it in the documentation, but it’s a > symbol:

   
   
   Click for full size

   Click that little button to display the panel.

4. The panel displays. Notice the caret changes to indicate you can close the panel:

   
   
   Click for full size

   Now, click the Twitter logo in the top left to return to the timeline.

5. The timeline displays:

   
   
   Click for full size

   I just tweeted something funny! Let’s see if anyone has picked up on it.

6. Click the Retweets link.
7. The Retweets menu displays. Click Your Tweets, Retweeted.
8. Your retweets display, but uh oh:

   
   
   Click for full size

   Notice that the caret is in the wrong position. To display the subpanel for this tweet, you have to click the caret twice. Once to get it into the proper state and the second to actually display the panel.

You have to have two things to work in QA: Wide eyes and quick eyes. These little tics are bugs and might not display on the screen for very long, so you have to see beyond where your mouse pointer or cursor is flashing. Additionally, a requirements document, if you even have one, probably isn’t going to outline precisely the state for this caret should be in each use case, so it’s probably not something you’re going to check off in your test checklist.

You just have to catch it.

Take a tweet, like this:

Click for full size

Now, say you have something clever to respond to the retweeter. Click Reply.

The reply box displays:

Click for full size

Note this is to reply to the person whose tweet was retweeted, not the retweeter. Well, that’s easy enough to fix.

Click for full size

In this case, I’ve replaced the reply with a sample (not to scale). Now click Tweet and….

Click for full size

Twitter tells me I’ve replied to the person whose tweet was retweeted, not to the value I to which I changed the @value.

Which sort of caused me a moment of panic when I thought I sent a senseless reply to someone I didn’t mean instead of an obscure reply to someone I meant.

Who would do that? I did it, didn’t I? The application should have recognized this possibility and either locked in the @name or picked up the @name or lack thereof for the success message.

Courtesy of CBS, we have a Never Happen happening:

In an effort to turn a popular Twitter feed into a broadcast comedy, CBS has given “$#*! My Dad Says” a rather un-DVR-friendly title.

Try to search for “$#*!” using your DVR’s remote control. Go ahead. We’ll wait.

It seems DVR designers quite understandably never suspected that a network would launch a TV show that started with the word “$#*!.”

Of course they would never do that until they did.

(Link seen here.)

Your organization probably trusts its third party integrated software partners as much as J.P. Morgan used to:

JPMorgan Chase is trying to move past three days of problems on its online banking site with an apology and an explanation that seems to put the cause on a third party.

The bank’s online site went offline Monday night and remained offline Tuesday. Service appeared restored by Wednesday, although there were some reports by Twitter users of problems.

The bank, in a statement posted online, said it was “sorry for the difficulties” that customers encountered, and said “we apologize for not communicating better with you during this issue.”

At first, Chase simply cited a “technical issue” for the problem. It has since provided a little more information.

The bank, the nation’s second largest, said in a separate statement that a “third party database company’s software caused a corruption of systems information disabling our ability to process customer log-ins to chase.com.” It added that the problem “resulted in a long recovery process.”

Now, how can you try to keep this from happening to you?

• Compel your vendors to tell you about their updates. Ideally, you would get a chance to test your software against their new versions before they promote them to production, but at the very least, they better tell you when they plan to put things up so you can test immediately. Remember, your “trusted” partners are organization filled with the same lying developer dogs as yours, but without the QA.

• Don’t do business with companies that practice continuous deployment. Seriously, they can promote at will and at whim, so your mission-critical software can fail at any time, without any warning, and without any clue that it’s not your fault.
• Run automated smoke tests against your production site as often as you can stand. Depending upon the nature of the application, this might only be daily, but the more frequently you can sanity check your production environment, the better. There’s nothing better than calling your head of development on Christmas Eve to tell him the site’s down before your users or clients even know.

Remember, you have no trusted partners. You should trust them even less than you trust your own organization, if you can imagine that.

CME says ‘sorry’ for dummy orders:

CME Group, the world’s biggest futures exchange, has said it is “deeply sorry” for mistakenly placing 30,000 fake orders it generated as part of a quality assurance test on its active energy and metals markets on Monday.

You know how easy that is to do, don’t you? Set a web configuration file wrong, put the wrong IP address in somewhere, and bam! your company loses $100,000. But your test passes!

Man, this is why I dealing with money or people’s safety in QA gives me fits. Therefore but the grace of luck go I.

I can only guess what browser the designers used when putting together this program.

IE? Probably not, since the Enter button lacks text:

Click for full size

Firefox? Well, maybe, if you discount the fact that the Enter button image was designed for use elsewhere:

Click for full size

Swell.

Hey, if you’re too rich, we don’t want you to enter:

Click for full size

Because none of our advertisers like households with incomes of $100,001 a year or more.

How does that confirmation e-mail display?

Click for full size

About as well as you would expect.

And what happens when you click submit without entering an e-mail address on the tell-a-friend form?

Sorry, I previewed that yesterday.

On the plus side, rest assured that this program was not as cheap to the client as it looks.