QA Hates You

So I recently decommissioned one of the PCs here in the lab. As part of it, I went through the Adobe process to decertify Dreamweaver on one machine, uninstall it, reinstall it on the new machine, and recertificate it. Then I created a shortcut of it on the desktop, and I ran it:

Oops.

Well, I wasn’t going to spend an hour or so on the phone trying to fix it, so I uninstalled it and reinstalled it on the new machine, and when I tried to run it again, I got the same catastrophic error message.

Was it time to call customer support and face a bullet of whatever precious metal Adobe uses? Of course not! I’ll take another look and….

Instead of creating a shortcut on the desktop, I’d copied the executable there and tried to run it, and the poor little application was lost in the big forest without its dependencies.

Yeah, there I was, a computer professional making a mistake that rendered Adobe Dreamweaver speechless.

So, what does your application do when it cannot find its dependencies? If it’s anything like this particular application, it spits up an unrelated error message and it asks you to call support. And support will have a werewolf of a time trying to figure it out.

So play around with your executable locations, including putting dependencies on network drives and not connecting to them, altering paths, maybe renaming directories without changing them–particularly if you can change the name of a directory from within the application and the application stores the hard file path somewhere within it (I used a testing tool once that let me do that, and then it could not find its own tests).

Granted, you’re not going to convince many people that these are high priority defects, but it’s something you should still consider when trying to build a quality application.

An Indiana resident reports some crazy swings in the weather this year:

Another reminder that your monitor is not the only output you need to test for your application.

(Seen here.)

An old Blockbuster envelope teaches us a valuable lesson about alternative methods of output:

Click for full size

So what portions of your application come out of the printer? Does it work right? Does it look right? Is it correct?

It’s not enough that you make sure the print dialog comes up correctly. You need to make sure that the extras that are often added to the printed page display correctly. For example, some maps add details such as the location, some Web sites put their names on it, and some applications use formula. To ill effect in this case.

If you want to be a real rapscallion, see what happens if you print to a file or to a PDF driver of some sort. Because someone out there in the real world just might.

Since yesterday’s New Twitter bug proved so popular, let’s look at another one. You can recreate this one on your own using the following steps in Mozilla Firefox:

1. Click the Retweets link.
2. The Retweets menu displays. Click Your Tweets, Retweeted.
3. Your retweets, assuming you’re some fraction as clever as I am and someone else repeated what you said, display. Highlight one and notice the little more info subpanel caret that displays. I don’t know what Twitterians call it in the documentation, but it’s a > symbol:

   
   
   Click for full size

   Click that little button to display the panel.

4. The panel displays. Notice the caret changes to indicate you can close the panel:

   
   
   Click for full size

   Now, click the Twitter logo in the top left to return to the timeline.

5. The timeline displays:

   
   
   Click for full size

   I just tweeted something funny! Let’s see if anyone has picked up on it.

6. Click the Retweets link.
7. The Retweets menu displays. Click Your Tweets, Retweeted.
8. Your retweets display, but uh oh:

   
   
   Click for full size

   Notice that the caret is in the wrong position. To display the subpanel for this tweet, you have to click the caret twice. Once to get it into the proper state and the second to actually display the panel.

You have to have two things to work in QA: Wide eyes and quick eyes. These little tics are bugs and might not display on the screen for very long, so you have to see beyond where your mouse pointer or cursor is flashing. Additionally, a requirements document, if you even have one, probably isn’t going to outline precisely the state for this caret should be in each use case, so it’s probably not something you’re going to check off in your test checklist.

You just have to catch it.

Take a tweet, like this:

Click for full size

Now, say you have something clever to respond to the retweeter. Click Reply.

The reply box displays:

Click for full size

Note this is to reply to the person whose tweet was retweeted, not the retweeter. Well, that’s easy enough to fix.

Click for full size

In this case, I’ve replaced the reply with a sample (not to scale). Now click Tweet and….

Click for full size

Twitter tells me I’ve replied to the person whose tweet was retweeted, not to the value I to which I changed the @value.

Which sort of caused me a moment of panic when I thought I sent a senseless reply to someone I didn’t mean instead of an obscure reply to someone I meant.

Who would do that? I did it, didn’t I? The application should have recognized this possibility and either locked in the @name or picked up the @name or lack thereof for the success message.

Courtesy of CBS, we have a Never Happen happening:

In an effort to turn a popular Twitter feed into a broadcast comedy, CBS has given “$#*! My Dad Says” a rather un-DVR-friendly title.

Try to search for “$#*!” using your DVR’s remote control. Go ahead. We’ll wait.

It seems DVR designers quite understandably never suspected that a network would launch a TV show that started with the word “$#*!.”

Of course they would never do that until they did.

(Link seen here.)

Your organization probably trusts its third party integrated software partners as much as J.P. Morgan used to:

JPMorgan Chase is trying to move past three days of problems on its online banking site with an apology and an explanation that seems to put the cause on a third party.

The bank’s online site went offline Monday night and remained offline Tuesday. Service appeared restored by Wednesday, although there were some reports by Twitter users of problems.

The bank, in a statement posted online, said it was “sorry for the difficulties” that customers encountered, and said “we apologize for not communicating better with you during this issue.”

At first, Chase simply cited a “technical issue” for the problem. It has since provided a little more information.

The bank, the nation’s second largest, said in a separate statement that a “third party database company’s software caused a corruption of systems information disabling our ability to process customer log-ins to chase.com.” It added that the problem “resulted in a long recovery process.”

Now, how can you try to keep this from happening to you?

• Compel your vendors to tell you about their updates. Ideally, you would get a chance to test your software against their new versions before they promote them to production, but at the very least, they better tell you when they plan to put things up so you can test immediately. Remember, your “trusted” partners are organization filled with the same lying developer dogs as yours, but without the QA.

• Don’t do business with companies that practice continuous deployment. Seriously, they can promote at will and at whim, so your mission-critical software can fail at any time, without any warning, and without any clue that it’s not your fault.
• Run automated smoke tests against your production site as often as you can stand. Depending upon the nature of the application, this might only be daily, but the more frequently you can sanity check your production environment, the better. There’s nothing better than calling your head of development on Christmas Eve to tell him the site’s down before your users or clients even know.

Remember, you have no trusted partners. You should trust them even less than you trust your own organization, if you can imagine that.

CME says ‘sorry’ for dummy orders:

CME Group, the world’s biggest futures exchange, has said it is “deeply sorry” for mistakenly placing 30,000 fake orders it generated as part of a quality assurance test on its active energy and metals markets on Monday.

You know how easy that is to do, don’t you? Set a web configuration file wrong, put the wrong IP address in somewhere, and bam! your company loses $100,000. But your test passes!

Man, this is why I dealing with money or people’s safety in QA gives me fits. Therefore but the grace of luck go I.

I can only guess what browser the designers used when putting together this program.

IE? Probably not, since the Enter button lacks text:

Click for full size

Firefox? Well, maybe, if you discount the fact that the Enter button image was designed for use elsewhere:

Click for full size

Swell.

Hey, if you’re too rich, we don’t want you to enter:

Click for full size

Because none of our advertisers like households with incomes of $100,001 a year or more.

How does that confirmation e-mail display?

Click for full size

About as well as you would expect.

And what happens when you click submit without entering an e-mail address on the tell-a-friend form?

Sorry, I previewed that yesterday.

On the plus side, rest assured that this program was not as cheap to the client as it looks.

What happens if your application doesn’t handle invalid e-mail addresses?

.NET handles it for you:

Click for full size

You really should never see this one, but I could say that about all of these, couldn’t I?

I’m really starting to fall out of love with Firefox. It’s becoming a resources hog, makes coming out of hibernation take a long time, and constantly doesn’t play nice with Flash these days. Additionally, I get this particular condition frequently:

Click for full size

This occurs if you have a large number of tabs open, taxing the browser, and then you use the keyboard to open the Bookmark menu (ALT, then B, then down arrow to start moving down it). Sometimes, it throws open the file menu and then leaves it open when you expose the Bookmarks menu.

On a severity scale, this is a low/cosmetic defect, but it just adds to the sense that the application is turning to crap.

How many cosmetic bugs does your organization think it can leave in a released application before the user thinks it’s crap? I bet the real number is far lower than your dev team thinks it is.

I found this hanging out of the receipt printer at a gas pump:

You know, if you’re copyrighting your single line test messages, maybe you’re overdoing it a little.

Does anyone want to guess whether this printer was untested or whether concerns about the inappropriate nature of this copyright were undiscovered or ignored?

I think I’ll go with tested, but undiscovered.

An error on the Comedy Central embedded media player:

I see an error message and a number. Is that an error enumerator?

Regardless, this is a troubleshooting message. This is not a message for the user. An error message to the user ought to include some sort of instruction to the user as what he can try to recover from the developer’s screw up.

But the lazy development program that allows these bugs is also the lazy development program that doesn’t bother to include good messages.

Check out this alert dialog box I got the other day:

It cannot believe that in 2010, someone is trying to install a Windows 3.11/Windows 95 compatible clip art browsing program that wants to use a 14,400 BPS modem to contact CompuServe on installation. I didn’t even tell it that I was a trained IT professional, which might have caused Windows Sockets to silently faint.

The 16-bit virtualization engine actually fainted dead away, which is why you cannot see its icon on the list of active taskbar items even though it’s there:

I would make some crack about the applications coming to me to die, but I’m the one who’s trying to get it to read punchcards, for crying out loud. It’s not handling the failure elegantly, though.

This does not represent a good practice of synchronizing your application data with the real world:

Twice we had ordered a pizza with extra-large pepperoni. Twice it had arrived without the extra large. See, the order is calibrated to hit everyone’s preferences, and my daughter will only have pepperoni, so: her half has pepperoni and extra-large pepperoni. But twice the pizza has arrived without.

“I’m going to call them,” I said.

“No, Dad, don’t! It’s okay! Don’t make a fuss about it.”

“Honey, a manager would want to know these things.”

So I called, and explained, and the manager asked if I ordered online. I said that I did, modern-type person that I was. That’s the problem. Extra-large has been discontinued, but it’s still on the online menu. Can you tell me what the printout on the bottom of the box said? I noted that it had elided the extra-large issue altogether. So the problem wasn’t on their end. [Emphasis added.]

This isn’t a simple change made on the fly, either. It’s a menu change determined probably by a national pizza chain’s HQ and telegraphed to its franchisees by semaphore or something. Somehow the change managed to dodge the people responsible for the Web storefront.

Forget keeping your application data synched with the other online data. Your application has to keep up with the real world, too. A lot of IT teams and vendors can rationalize not keeping up if the customer doesn’t keep up, but you need to make it easy to change and to grab your client/internal stakeholders by the lapels to ensure they keep you up to date.

James Lileks, from whose blog I took the anecdote, is a patient customer and does not blame his local pizza shop. However, another client will quit a brand for that sort of thing. Especially if that client is in QA.

So the local newspaper, the Springfield News Leader, has a little problem with its site. When you view an article that spans multiple pages, the page numbers or the next page link does not work in Firefox. This is quite the reverse of most software, which typically works in Firefox or Chrome but bombs out in Internet Explorer and the developers sniff about the hoi polloi and their attachment to the dominant Internet browser.

So I go to the send feedback form, fill out a defect report, and….

Click for full size

That’s what I get, I suppose.

In a stunning turn of events, the problems remain unfixed.

So I stopped by the Branson (Missouri) Regional Airport recently, and I spotted this kiosk:

Click for full size

It offers the user the opportunity to enter some sort of contest to go to Nashville. It’s obviously a Web browser in kiosk mode, but this one has a full keyboard with a trackball and two mouse buttons. Uh oh.

So I click the Contest Rules link at the bottom and get the contest rules, which has a naked link at the top that takes you back to the form. But hover over the link and right click and…. Uh oh.

Click for full size

What happens if I open that in a new window? Hello, Internet!

Click for full size

So a user has complete access to the Internet. Go where you want. Get all the malware you want. I didn’t try to see if a regular download and install worked, but I would not doubt it. What happen if I ALT+TAB?

Click for full size

Lookie there! Lookie there! It’s the command line. A little CTRL+C action and I have access to issue commands to the machine and maybe even the network.

So is that Cat-5 cable running out of the back of the box connected to the airport network itself or a dedicated safe portal to the Internet? Given what we’ve seen here, what do you think?

If you’re ever called to check out a kiosk application, not only should you run through the form the kiosk will host, but you should get a kiosk itself and run it through its paces and look outside the confines of the application to look for security pitfalls.

You need to check out the user interface action. This kiosk gives the user all the normal tools that users need for full input opportunity to the Internet. Some kiosks only have touchpads or touchscreens. Here are a couple of things to think about:

• Know your keyboard shortcuts. Most people don’t know these keyboard shortcuts, but they do things to your active window (even your kiosked browser). What can you do with that?
• Know your internal browser behavior. I remember seeing a kiosk with only a touchscreen that offered the Web sites of a building’s residents. Within a touchscreen environment, you would think you’re limited to navigating through links in the browser window. You would be wrong. mailto: links trigger the helper application associated with e-mail. What can you do when you try that?
• What happens when you unplug the machine and plug it back in? It reboots, probably, affording you the ability to go into alternate bootup scenarios and whatnot. Should your user have access to that? Probably not.

To begin vetting kiosks, you need to think outside the terms of your application and think in terms of the technologies that encapsulate it. The better you understand those and can identify the ways users could interact with the whole kiosk, the better you can prevent them from doing so inappropriately.

Isarian sends along another episode of a media player that displays NaN when it doesn’t know when a program will end:

Click for full size

Come on, guys, is it so hard to think up or implement a conditional that says IF this is a live stream THEN do not show the length of the clip?

Geez, I think I just accidentally did a little freelance software architecture or something. It hurt. No wonder the actual people tasked with the responsibility avoid it, too.

(Previously on QAHY….)

The phrasing on this survey question stopped me in my tracks:

Click for full size

The label says you should use the number 5 for the one you use the most and the number 1 for the one you use the least. Then it gives you 6 items.

Really, I think the survey writers want you to rate the frequency, but they use superlatives instead of comparatives here, so logically you cannot rank them from the least to the most without leaving one out.

At least not to my orderly mind. Maybe Joe has a solution. His mind is less highly developed, which is why he would not be taking a survey about the Green Bay Packers in the first place.

When you have an intersection of public money – software project – no testing, you’re achieving a singularity of gonnagobadly.

Latest example: Queensland (Australia) Health’s new payroll system.

Look at this recipe of success with its pinch of simpleton’s drive for simplicity and its dash of “deadline-at-any-cost.”

Among the board’s decisions was to change the definition of severity one and severity two defects so the project could pass exit criteria.

During testing, the board decided not to undertake a full parallel pay run test because of the size and complexity of the task.

In January, the testing company suggested either the rollout be delayed until a full system and integration test was done, or the board accept that untested scenarios might not go to plan.

The board chose to accept that risk over delaying the rollout.

You can guess how well that went given it’s in CIO magazine.

On the plus side, it was only $24 million over budget, only 60%. Here in the United States, you don’t get enough code written for government projects with a mere 60% overrun to test at all.

(Link seen on Cartoontester’s Twitter feed.)