Software Testing: Art or Science?

When I was growing up in the 70′s I got an electric guitar for Christmas.  I had dreams of becoming the next Eric Clapton or Jimmie Page.  (For you youngsters that have no clue who I’m talking about – Google them).  I took lessons and played until my fingers bled.  But no matter how hard or how long I played – I was no Eric Clapton.  Oh I could play the notes, but not the music.  I still play to this day….some 35 years later, but it is still more mechanical than music.  So here I am writing a software testing blog.   :0)

I digress (as usual) – back to the question at hand.  Is software testing an art or a science?  The simple answer – Yes!

Testing – good testing – is both art & science.  I’ve seen, read, and heard the arguments on both sides and I really can’t say one is any superior to the other.

From the scientific side, testing can include a number of proven techniques such as equivalency class testing, boundary value analysis, pair-wise testing, etc.  These techniques, if used properly, can reduce test times and focus on finding the bugs where they tend to hang out – much like a porch light on a summer night.  The scientific techniques are all easily learned and applied.  They are also (or should be) well documented and easily repeatable with the next build to help validate defect resolutions.  Personally, I can’t think of any testing situation where some level of scientific testing is not needed.  If you are testing critical sytems or systems where life or death is a likely outcome (air traffic control systems, medical systems) – these tests are unavoidable.  Does that mean there is no need for other, more cranial type tests?  Absolutely not! 

But there is an art to testing as well.  The art side is not as easily learned.  It comes from experience, from gut feelings, from all those areas that you can’t really document.  It is typically those tests that start with thinking “I wonder what would happen if…..”  Exploratory Testing is a great example of the “art” of testing.  It involves poking around an application – with a purpose, trying to find those elusive bugs – guided only by gut instinct.

I can teach you how to play the notes, but I can’t make you a musician.  That my friends is something inside you that you must cultivate yourselves.  How do you get to Carnagy Hall?  Practice, Practice, Practice!

To everyone in the US – Happy Thanksgiving!!  To everyone else – Happy Thursday!!  Keep those cards and letters (or emails and blog comments) coming.  I love to hear from you!

The Angry Tester

I just got finished reading through the blog of one of the “Test Celebrities”.   You know the ones I’m talking about – they write the books, speak at all the conferences, etc.   I have to admit, I don’t like many of them.  I went to a conference once and watched three of them gang up on a poor speaker.  They all share one thing in common – believe as they tell you or else.  How dare anyone disagree with them?  Well I do, often, and I’m saying it here!

My experience also hit home – I never want to be one of them!  I’ll give you my 2-cents worth here, but it is just that….my opinion.  Take it or leave it.  It really doesn’t matter to me.  Please disagree with me – I encourage it!  I’m not the expert in software testing, nor have I or will I, ever claim to be.  Unlike my “colleagues” I would never publically humiliate a speaker.

Anyway – I digress.   This “Test Celebrity” is very angry.  There were two particular hot strings to follow.  The first is an ongoing feud between himself and another celebrity.  The second – someone had the audacity to question the expert.  How dare they!  The really sad part is that they seem to have followers who are just as angry, if not more so.

I disagreed with one of them once.  I was following a software testing mail list and made a comment (nothing  derogatory) about a task they were giving to the members of the list.  Basically they presented a test scenario and asked the members to write in describing how they would test it.   I had seen this tactic before.  They would typically provide a nasty response back to the group if anyone dared to think differently.  I wrote back simply saying that it might be a helpful learning experience if the “celebrity” told the group how he would handle it using their process.  The next day one of  the other “celebrities” added a post to the group publically ridiculing ME! Again, I’m open to disagreement and I have no problem if you disapprove of me or my ideas in a public forum.  But this was just downright nasty!  Something that would have best been delivered in a private email and not a public post.  He was also one of the culprits that ridiculed the conference speaker.  Needless to say, I have absolutely no respect for him or his ideas.

So who do I like?  Very few.  I can count them on one hand.  I read their stuff (books, articles, etc.) all the time and I’ve implemented some of their ideas.  They are not only entertaining – they know their stuff.  I try to never miss a chance to see them.  Three speakers to never miss:  Randy Rice, James Whitaker, and Lee Copeland.  The “always miss” list has 4 members, but I’ll keep that one to myself.  I won’t seek them out, but if they are at a conference I’m attending I’ll stop by for a visit.  Payback is a b%^&*!    GAME ON!   )

I Hate Agile!

OK…before you start the hate mail…maybe hate is too strong a word.  Actually it is not the Agile (or Scrum, or XP) process that I hate, but the Agile practiotioners are really starting to get to me. 

I just finished reading about how one practitioner (I’m not going to name anyone specific….you know who you are) is upset that some newly converted groups tend to fall off the Agile wagon, back into old habits.  He chose of course to publically ridicule them as a group (thankfully, no specific names).  As with most of these “born-again” Agile cultists, if you don’t strictly adhear to their rules you are no longer worthy of their support, and are subject to public condemnation.  Agile has become a religious cult and the practitioners have become like early American religious fanatics (the ones kicked out of England for their fanatic beliefs).  “Believe as we do or we shall excommunicate you from the flock!”   Well then point me to the pillory – I confess – I’m a non-believer!

I wish there were one single, out-of-the-box development process that would work universally for everyone.  There isn’t!  There never will be!  Personally, I like to follow what I call the “Gumbo” process.  A little bit of this, a little bit of that, add what works, subtract what doesn’t – heat and stir and Voila! – Software (or as Eddie Izzard would say: “Hootcha, hootcha, hootcha….software). 

Don’t get me wrong – Agile has some really good ideas, but so does Iterative, RUP, and believe it or not – Waterfall!  That’s right – I said Waterfall – rack me!  Hand me my scarlett letter – I will wear it proudly!

Will They Ever Learn?

Strike two….

http://www.usatoday.com/travel/flights/2009-11-19-faa-computer-delays_N.htm

Is there a risk to not adequately testing….maybe we should ask the FAA!

Data, Data, Data!

I seem to have a thing for data issues lately.  Here’s the latest….

My son, Christopher, drives a Dodge Ram 4×4 pick-up truck.  He recently registered online with the local Dodge Truck owner’s club.  The registration process went smoothly, however, when he logged into the site he received a “Welcome Christophe” message.  Needless to say – he was $%&&ed!  It happens to him all the time.  He could just use “Chris” but nooooo!  Sadly I have seen this problem too many times!

The source of the problem:  the database, input form, and output screen are out-of-sync.  My guess is that the field size was never defined which allowed the user to enter a First Name with no constraint on field size (unlimited length).  The database field was probably set to 10 characters and truncates anything greater than 10 (with no error message).  The UI then displays whatever is in the database (bad data).  There was no input validation (yes, I tested it).  So when the name is displayed – only the first 10 characters are shown.  He was born in Germany and my wife wanted to name him the German form “Christoph”.  I objected, and now I’m paying for it.

But this drives home the need to include Test/QA during the requirements and/or design process.  Would I have caught this in requirements review?  I would.  Would you?

This is an easily avoidable problem.  First we could probably increase the field size in the database.  But is that enough?  Not to my thinking.  If we are going to use the field in processing downstream (like displaying it).  I’m probably going to ask for a limitation in the data entry form in the UI, and validation with appropriate error handling.  And make sure they are in sync!  If the database only allows 10 characters, I’m going to look for a 10 character limitation and error handling in the UI.  BTW – I’m also going to object to the 10 character field length. 

If I can catch this in a requirements review or a code review it is a relatively easy and cheap fix.  If you wait until testing it’s gonna cost ya!

#$%& Windows 7!!!

OK I have to admit it – I really like Windows 7.  Installing it, however, is a completely different story.

I’m not a fan of the installation or upgrade process.  It took me 3 days to get my copy of Windows 7 Ultimate running.  THREE DAYS!  I spent 1 day alone on the phone with customer service.  Totally unacceptable.

Sometimes I wonder how the boys and girls in Redmond come up with these ideas!

So here’s the story…..

As a Microsoft Partner and a Microsoft Action Pack subscriber, I got Windows 7 Ultimate and Professional as part of my subscription.  When I attempted to install it on my 4 year old Dell laptop, I found that the free space on my hard drive was severely limited and I needed to do some upgrades to run Windows 7.  So I upgraded my memory to the 2 GB max, bought a new 360 GB hard drive (which Dell assured me was compatible), and upgraded my CD ROM to a CD/DVD ROM.  Everything worked fine except the hard drive.  I was already at the maximum hard drive size for my laptop – 40 GB.  The 360 GB drive was not compatible after all. Dell was really good about it and allowed me to return the drive with a full refund and they paid shipping.  A minor inconvenience.  I bought a 1 TB external hard drive which I plan use it for most of my storage needs.  I had to remove a couple of seldom used applications, but again – no biggie.  I was ready – or so I thought. 

After a few minor issues, I installed Windows 7 Ultimate on the Dell.  When I launched it for the first time, the first thing I noticed was really low screen resolution.  No big deal I thought, just increase the screen resolution.  So off I go to the Device Manager where I find the video is set to Standard VGA – the default – and it was set at maximum resolution.  Again, no big deal.  Frustrating but no biggie.  I just need to install the updated video driver.  Unfortunately, since my 4 year old laptop was considered “end of life”, no updated drivers were available.  Again, frustrating, but in the back of my mind I was afraid this would happen.  Time for a new laptop.  I begged and pleaded with my wife until she finally relented and let me buy a new one.  So off I went, like a kid at Christmas to buy a new toy. 

I find the laptop I want, at a really good price.  It had lots of memory and was expandable to a bunch more.  It came with a huge hard drive, really high resolution video, high quality sound, a built in web cam – the works.  The best part – it came with Windows 7(Home) already installed.  A simple upgrade to Windows 7 Ultimate and I’m back in business!  Here’s where the fun starts.

The next day we had a major snowstorm and the office was closed.  Cool – a whole day to play with my new toy.  Step 1 – upgrade Windows 7 from the Home version to the Ultimate version.  The first glitch – you can’t upgrade from the Home version to the Ultimate version (or the Professional version either for that matter).  I was shocked!  Why wouldn’t Microsoft let me upgrade to a higher version!?  If a customer wants to pay the money and upgrade to a higher version of your product, why wouldn’t you let them? I would have love to have been in the room when they worked that one out!  So I was forced to do a clean install of Windows 7 Ultimate which of course wiped out all of my existing files to include the device drivers.  I do the clean install and start it up.  Again, low resolution video.  No biggie – I’ll reinstall all the device driver files that came on the DVD included with my new laptop.  Nothing came with the laptop. I’m getting a little frustrated, but still keeping my cool.  I went to the vendor’s website to find the drivers – nothing for Windows 7 (well nothing obvious anyway).  Crank up the frustration.  I called the vendor’s customer service line.  After being passed to 3 different people I was finally told to contact Microsoft.  Crank it up a bit more.

So I call Microsoft Customer Service (and I use that term loosely) where they promptly tell me to call the vendor.  I told them I called the the vendor first and they told me to call you.  “Just a minute sir, I’m going to put you an hold for a minute.”  Like I have a choice.  After 10 minutes on hold, I am passed off to a customer service rep who is obviously reading from a customer service manual.  After 30 minutes or so she declares that my device drivers appear to be missing or out of date.  A blinding glimpse of the obvious!  Again, I’m put on hold.   15 minutes pass this time until someone finally picks up.  Once again I explain the whole scenario, only to find out this person I’m talking to is in marketing and has no clue what I’m talking about.  I’m put on hold again, this time when someone picks up I find I’m talking to sales.  You have GOT to be kidding.  I have now gone from frustrated to downright p%^&ed off!  I was passed off to a few others who had no clue how to help me.  They eventually sent me to Technical Support.  Finally!  Someone who speaks technical.  Right!  Not so much.

I get to Technical Support where they assure me they will not let me off the phone until I am completely satisfied that the problem is fixed.  Only 1 little thing – it was going to cost me like $200!   That is where I lost it.  “You want me to pay $200 to be able to use your product!”  At this point I’m thinking return the laptop and do like the commercial says and buy a Mac!  Which I not-so-politely to the person on the phone.  “Hold please!”

The next person on the phone is from Networking I think.  Definately a technical person!  For like the 10th time I retold my story.  I also reminded him there was absolutely NO WAY I’m paying a dime!  No problem he said.  Within 10 minutes he had found the driver set I needed, walked me through the download and installation, and stayed on the phone with me until my new laptop was finally up and running.  Whew, I’m going to bed.  So much for my day off!

Day three – With Windows now working I start reinstalling my applications.  I reinstall Office, transfer over all of my documents, pictures, and other personal files, and set up my email in Outlook.  Everything is going fine – until I get to Microsoft Project.  It tells me I can’t install it because it can’t find an installed version of Office.  I verify that I did indeed install Office.  I did and it appears to be working fine.  It must be a glitch I thought…so restart and try to reinstall Project.  Same error.  That’s it – I’m so done!

So moral of the story – three weeks later, I’m finally back to where I was with the exception of Project.  I called Microsoft, but again they want to charge me $200 to fix the problem, to which,  I’m still adamant that I WILL NEVER PAY!

As a QA person and life-long quality advocate this is TOTALLY UNACCEPTABLE!  I’m guessing they don’t want me for one of their Windows 7 commercials.

Where are my heart pills!

Severity vs. Priority

Synonyms?

Some would say yes.  I scream NOOOOOOOOO!

Severity and Priority are really two completely different concepts when it comes to managing defects.  Let’s take a quick look:

Severity defines the impact that a given defect has on the system.  A severe defect may cause the system to crash or invoke the dreaded blue screen of death.  I don’t think anyone would argue about its severity.  But how did it happen?  Did it take an obscure set of key strokes or does it happen anytime the uses presses “e”?   Or does it only happen only in the month of June, during sunspot activity?  What about a spelling error?  Or maybe the text color is a really annoying neon green.  Severe?  Probably not.  Maybe just a cosmetic issue.

Priority, on the other hand, defines the order in which we should resolve a defect.  Should we fix it now, or can it wait?  How difficult is it to resolve?  How many resources will be tied up by the resolution?  Let’s look at our two previous examples.  Issue 1 (system crash) is definitely severe, may be difficult to resolve, but only happens rarely.  When should we fix it?  Contrast that with the second issue (spelling error).  Not severe, just makes you look bad.  Should be a real easy fix.  One developer, maybe 10 minutes to fix, another 10 to validate (if that).  Which should get the higher Priority?  Which should we fix now?  I’m going to recommend fixing the typo immediately, and if there is sufficient time, fix and resolve the blue screen before the next build.  I will probably wait until the next major release.  It becomes….a Release Note!

Many commercial defect tracking systems have either Severity or Priority. Some may have both.  Sadly, I know of only one.  But others allow you to modify existing fields or add additional fields.  Of course those are severely lacking in other areas.  The perfect defect tracking system does not exist – yet!

Bottom-line: you need to define both Severity and Priority for your application, and based on your user’s needs. Rarely is an “out-of-the-box” solution adequate.

BTW – To solve this dilemma, I am building Dave’s Perfect Defect Tracking Tool, code-named Unicorn.  Why “Unicorn” you ask?  Because, like unicorns, I have heard they exist, most people can tell me what one looks like, but I’ve never seen one.  Unicorn will be fully compatible with Dave’s Perfect Test Management System, code-named “Bigfoot”. Stay tuned!

Welcome!

Welcome to my blog!  People have been asking me to create a blog for some time now, so I figure it is time to get off my lazy butt and do it.  So here it goes….

Hi! – my name is Dave Whalen.  I am a self-proclaimed “Software Entomologist” (much like Michael Jackson called himself the “King of Pop”).  Which essentially means I study, and have dedicated my life to, software testing and bugs.  I have been testing both hardware and software (sometimes both at the same time) for over 20 years.  These days I am an independent software testing consultant – proud owner of “Whalen Technologies”.  I currently live in Denver, Colorado where, as I write this, it’s @#$%ing cold!  I ocassionally speak at local and national software testing conferences (especially if they are held in Las Vegas).  My specialties are testing processes and procedures.  I’m currently focused on Defect Management processes and Tools.  I’ll try to post something at least weekly.  More often if I have the time or a particular subject to rant about – which I typically do.  Feel free to comment, or contact me, or send me your favorite Gumbo recipe!