QA Hates You

I’m really starting to fall out of love with Firefox. It’s becoming a resources hog, makes coming out of hibernation take a long time, and constantly doesn’t play nice with Flash these days. Additionally, I get this particular condition frequently:

Click for full size

This occurs if you have a large number of tabs open, taxing the browser, and then you use the keyboard to open the Bookmark menu (ALT, then B, then down arrow to start moving down it). Sometimes, it throws open the file menu and then leaves it open when you expose the Bookmarks menu.

On a severity scale, this is a low/cosmetic defect, but it just adds to the sense that the application is turning to crap.

How many cosmetic bugs does your organization think it can leave in a released application before the user thinks it’s crap? I bet the real number is far lower than your dev team thinks it is.

I found this hanging out of the receipt printer at a gas pump:

You know, if you’re copyrighting your single line test messages, maybe you’re overdoing it a little.

Does anyone want to guess whether this printer was untested or whether concerns about the inappropriate nature of this copyright were undiscovered or ignored?

I think I’ll go with tested, but undiscovered.

An error on the Comedy Central embedded media player:

I see an error message and a number. Is that an error enumerator?

Regardless, this is a troubleshooting message. This is not a message for the user. An error message to the user ought to include some sort of instruction to the user as what he can try to recover from the developer’s screw up.

But the lazy development program that allows these bugs is also the lazy development program that doesn’t bother to include good messages.

Check out this alert dialog box I got the other day:

It cannot believe that in 2010, someone is trying to install a Windows 3.11/Windows 95 compatible clip art browsing program that wants to use a 14,400 BPS modem to contact CompuServe on installation. I didn’t even tell it that I was a trained IT professional, which might have caused Windows Sockets to silently faint.

The 16-bit virtualization engine actually fainted dead away, which is why you cannot see its icon on the list of active taskbar items even though it’s there:

I would make some crack about the applications coming to me to die, but I’m the one who’s trying to get it to read punchcards, for crying out loud. It’s not handling the failure elegantly, though.

This does not represent a good practice of synchronizing your application data with the real world:

Twice we had ordered a pizza with extra-large pepperoni. Twice it had arrived without the extra large. See, the order is calibrated to hit everyone’s preferences, and my daughter will only have pepperoni, so: her half has pepperoni and extra-large pepperoni. But twice the pizza has arrived without.

“I’m going to call them,” I said.

“No, Dad, don’t! It’s okay! Don’t make a fuss about it.”

“Honey, a manager would want to know these things.”

So I called, and explained, and the manager asked if I ordered online. I said that I did, modern-type person that I was. That’s the problem. Extra-large has been discontinued, but it’s still on the online menu. Can you tell me what the printout on the bottom of the box said? I noted that it had elided the extra-large issue altogether. So the problem wasn’t on their end. [Emphasis added.]

This isn’t a simple change made on the fly, either. It’s a menu change determined probably by a national pizza chain’s HQ and telegraphed to its franchisees by semaphore or something. Somehow the change managed to dodge the people responsible for the Web storefront.

Forget keeping your application data synched with the other online data. Your application has to keep up with the real world, too. A lot of IT teams and vendors can rationalize not keeping up if the customer doesn’t keep up, but you need to make it easy to change and to grab your client/internal stakeholders by the lapels to ensure they keep you up to date.

James Lileks, from whose blog I took the anecdote, is a patient customer and does not blame his local pizza shop. However, another client will quit a brand for that sort of thing. Especially if that client is in QA.

So the local newspaper, the Springfield News Leader, has a little problem with its site. When you view an article that spans multiple pages, the page numbers or the next page link does not work in Firefox. This is quite the reverse of most software, which typically works in Firefox or Chrome but bombs out in Internet Explorer and the developers sniff about the hoi polloi and their attachment to the dominant Internet browser.

So I go to the send feedback form, fill out a defect report, and….

Click for full size

That’s what I get, I suppose.

In a stunning turn of events, the problems remain unfixed.

So I stopped by the Branson (Missouri) Regional Airport recently, and I spotted this kiosk:

Click for full size

It offers the user the opportunity to enter some sort of contest to go to Nashville. It’s obviously a Web browser in kiosk mode, but this one has a full keyboard with a trackball and two mouse buttons. Uh oh.

So I click the Contest Rules link at the bottom and get the contest rules, which has a naked link at the top that takes you back to the form. But hover over the link and right click and…. Uh oh.

Click for full size

What happens if I open that in a new window? Hello, Internet!

Click for full size

So a user has complete access to the Internet. Go where you want. Get all the malware you want. I didn’t try to see if a regular download and install worked, but I would not doubt it. What happen if I ALT+TAB?

Click for full size

Lookie there! Lookie there! It’s the command line. A little CTRL+C action and I have access to issue commands to the machine and maybe even the network.

So is that Cat-5 cable running out of the back of the box connected to the airport network itself or a dedicated safe portal to the Internet? Given what we’ve seen here, what do you think?

If you’re ever called to check out a kiosk application, not only should you run through the form the kiosk will host, but you should get a kiosk itself and run it through its paces and look outside the confines of the application to look for security pitfalls.

You need to check out the user interface action. This kiosk gives the user all the normal tools that users need for full input opportunity to the Internet. Some kiosks only have touchpads or touchscreens. Here are a couple of things to think about:

• Know your keyboard shortcuts. Most people don’t know these keyboard shortcuts, but they do things to your active window (even your kiosked browser). What can you do with that?
• Know your internal browser behavior. I remember seeing a kiosk with only a touchscreen that offered the Web sites of a building’s residents. Within a touchscreen environment, you would think you’re limited to navigating through links in the browser window. You would be wrong. mailto: links trigger the helper application associated with e-mail. What can you do when you try that?
• What happens when you unplug the machine and plug it back in? It reboots, probably, affording you the ability to go into alternate bootup scenarios and whatnot. Should your user have access to that? Probably not.

To begin vetting kiosks, you need to think outside the terms of your application and think in terms of the technologies that encapsulate it. The better you understand those and can identify the ways users could interact with the whole kiosk, the better you can prevent them from doing so inappropriately.

Isarian sends along another episode of a media player that displays NaN when it doesn’t know when a program will end:

Click for full size

Come on, guys, is it so hard to think up or implement a conditional that says IF this is a live stream THEN do not show the length of the clip?

Geez, I think I just accidentally did a little freelance software architecture or something. It hurt. No wonder the actual people tasked with the responsibility avoid it, too.

(Previously on QAHY….)

The phrasing on this survey question stopped me in my tracks:

Click for full size

The label says you should use the number 5 for the one you use the most and the number 1 for the one you use the least. Then it gives you 6 items.

Really, I think the survey writers want you to rate the frequency, but they use superlatives instead of comparatives here, so logically you cannot rank them from the least to the most without leaving one out.

At least not to my orderly mind. Maybe Joe has a solution. His mind is less highly developed, which is why he would not be taking a survey about the Green Bay Packers in the first place.

When you have an intersection of public money – software project – no testing, you’re achieving a singularity of gonnagobadly.

Latest example: Queensland (Australia) Health’s new payroll system.

Look at this recipe of success with its pinch of simpleton’s drive for simplicity and its dash of “deadline-at-any-cost.”

Among the board’s decisions was to change the definition of severity one and severity two defects so the project could pass exit criteria.

During testing, the board decided not to undertake a full parallel pay run test because of the size and complexity of the task.

In January, the testing company suggested either the rollout be delayed until a full system and integration test was done, or the board accept that untested scenarios might not go to plan.

The board chose to accept that risk over delaying the rollout.

You can guess how well that went given it’s in CIO magazine.

On the plus side, it was only $24 million over budget, only 60%. Here in the United States, you don’t get enough code written for government projects with a mere 60% overrun to test at all.

(Link seen on Cartoontester’s Twitter feed.)

I’ve lamented on Twitter recently that the innate ability to break software is as much a curse as a blessing. When I’m trying to use software, it breaks.

Case in point: I went out and bought Adobe Acrobat 9, shelling out the big $250 to support a project that will net me only a little more than that, and when I tried to use it on the single file I needed to modify, Adobe Acrobat crashed to the desktop without a by-your-leave. You bet I sent those crash reports to Microsoft automatically so they could snicker about Acrobat.

Secondly, I lassoed a couple of files on the desktop and dragged them to the recycle bin, and they were no longer available to delete, but were caught between the dimensions like Captain Kirk in The Tholian Web. They flickered in the corner of the screen regardless of what application covered the desktop:

Click for full size

I had to restart to clean them up.

Is it that I am just that good, or is it that software sucks in 2010? More plaintively, I raise my eyes to the heavens and ask, “Why does the That Will Never Happen keep happening to me?”

The TroyBilt Web site, particularly the part that allows you to buy parts for your TroyBilt outdoor maintenance equipment, drove me crazy when I tried to use it.

Here’s just one of many things awry:

Click for full size

Note the pattern provided by the phone number. In some phone numbers, parentheses go around the area code here in the United States. Because I am that way, I put parentheses around the whole thing to see what would happen. It gave me the above error message.

I fixed it. Then, as I was trying to opt into the newsletter to take a look at it and see if it was worthy of mockery, I entered an e-mail address and a password. The application returned new error messages from the server-side to tell me I needed a number in the password. So I cleared it out and tried again, but because it was server-side, it abandoned my credit card information, so I had to type that back in. Finally, I successfully ordered an out-of-date part I found looking at exploded diagrams of my new tiller which I rendered inoperative within 10 minutes of firing it up. The listing for the part told me the part was out of date, but I’ll be hung if I was going to spend any more time fishing around on the site to find it since it did not provide me a link to the updated part.

Gah. I feel bad for professionals who have to deal with that particular utility daily.

The state of Missouri has put a form on the Internet so that teenagers can anonymously report bullying behavior in their schools. As a normal piece of government work, it’s a bit of a slapdash form, but it does feature the following JavaScript validation message that matches one in our defect tracker:

Click for full size

Strangely, though, the state of Missouri’s form doesn’t have the option I select most: Ultra.

In New York City, some test data made it into production. Unfortunately, it was a police system and led to police raiding a couple’s home fifty times:

The snafu started in 2002, when police used the Brooklyn address as part of what Browne called “random material” to test an automated computer system that tracks crime complaints and records of other internal police information. Before that, the work was done manually.

The couple first complained about the harrowing police visits in 2007, when Rose Martin wrote a letter to Kelly. “And we identified the problem then,” Browne said. “It was a mistake by the police department.”

So what do you do to manufacture test data with the foresight of knowing that it could go into production? You could make it obvious, such as “Blah blah blah test,” but then it looks silly when that leaks up.

I, on the other hand, prefer to use a set of standard user names that include non-obvious markers such as using the company’s name as the last name. For the address, I always use the business address or the CEO’s home address.

I’m not keen on playing with a snapshot of real people’s data because something like this can happen. And if it’s the CEO’s door getting banged on by the SWAT team, you can bet it won’t take 50 visits to get it fixed.

(Link to this story seen in the Twitterverse, but I was tracking the story from before that because this sounded like a bad data issue even without the confirmation. Not to mention a poor process on the part of the police to require a fiftieth bad raid to make them fix it.)

Hey, is that another Facebook bug?

Last night, in an embarrassing glitch for Facebook that raises questions about privacy on the site, some users of the social-networking service began getting hundreds of personal messages that weren’t intended for them.

A WSJ.com editor, Zach Seward, tipped Digits off to the apparent glitch after his Facebook inbox was flooded with messages ranging from the mundane to the truly private.

“I am sorry for letting my jealousy and worry get the best of me,” reads one of the emails.

Oh, boy, Facebook is really in the race to obsolescence here. Misdirected personal messages? That’s going to undermine user trust and confidence.

And once one of the other up-and-coming social networking solutions hits a tipping point, Facebook is going to lose users in droves. You can’t keep burning people with poor quality and expect to be an ongoing concern no matter how many Farmville players you have.

Sobe’s new contest misses it by that much:

Click for full size

“In the 10 Ring,” he explained to his foreign readers, refers to hitting a bullseye when target shooting.  Sobe missed the frame slightly here when dropping their Flash application onto their Facebook page.

Want to know what else they did wrong with the contest entry?

• They put the rules in a pop-up window which Internet Explorer blocks since it’s coming from a Flash application.  Instead, they could have put it in a panel in the Flash application.
• They ask for a phone number, but they don’t tell you what a proper phone number is.  You get to try and err.
• They don’t enable the Submit button on the Tell a Friend form unless you enter valid e-mail addresses.  The other steps, though, enable the Next/Submit buttons before the user has filled out the form.

Well, it was good enough to separate Sobe from its interactive budget.  Carry on, then!

The State of Missouri in the United States has uncovered a computer error in its favor:

 Missouri acknowledged Monday that it reported inflated numbers of food stamp recipients to the federal government, calling into question millions of dollars of bonuses paid to the state for running one of the nation’s top-flight programs.

The Department of Social Services said a computer programming error has consistently exaggerated the figures submitted since September 2002.

You know, the most cynical amongst us would claim that computer errors that “accidentally” would give preferential treatment to the organization using or writing the software would get qualified as lower priority defects and would be allowed to run as long as possible.

However, the teams I’ve worked on probably wouldn’t allow that sort of thing to occur.  But maybe I’m not cynical enough.

I imagine that’s what happened here: Programmer slip-up produces critical bug, Microsoft admits:

“Look at the two array references to ValidateRoutines[] near the end,” said Michael Howard, principal security program manager in Microsoft’s security engineering and communications group, referring to a code snippet he showed in a post to the Security Development Lifecycle (SDL) blog. “The array index to both is the wrong variable: pHeader->Command should be pWI->Command.”

One can imagine that developer, cranking along bopping his head to some Shakira, coming to the place to insert a variable name in the code, typing a couple letters and then letting the IDE autofill the wrong variable name.  A code review could catch that, I suppose, if you were diligent enough, but who has time for code reviews when there are deadlines approaching and no time to squander on anything but Rock Band Beatles in the lounge?

Like most everyone else, Microsoft has run into problems with localizing its Web site from English to Polish.  For example, Engadget finds an instance where someone altered, badly, a stock photo instead of getting a new one.

Poking around the site, I found a couple additional flaws.

(more…)

Found on a box:

Uh huh.

You know, if you’re ever called upon to test label printers, scanners, or any of the various and sundry other peripherals your software will use or interact with, you get those printers, scanners, or peripherals and test on them.  Don’t trust the developers and whatever emulators they might have concocted.  Then, build a complete set of test cases for the apparatus.  Then, run those test cases.  Finally, make developers fix the problems you find.

Sure, it’s not as sexy as the main module of your shipping software or the Web interface or the other fun things developers want to do.  However, it is the basic thing your software needs to do.  It’s what your users expect it to do.  And it better do it.