Is Automation Always The Answer? Not So Fast…

When I was a teenager – many years ago – Santa bought me a new set of tools.  I had my license and an old used Toyota.  Santa apparently thought I needed tools.  It was a really nice auto mechanics tool set, complete with a really nice tool box.  Not to seem ungrateful, but I knew absolutely nothing about auto mechanics.  40-some years later, I still don’t.  I fixed electronic stuff like radios, TVs, and stereos – still do.  Mostly computers these days.  Even if I had the first clue about auto mechanics – I drove a Toyota.  The tools were standard, I needed metric.  With the exception of the set of screw drivers, I couldn’t do much with it.  In fact, I could only really use 3 of the 12 screw drivers in the tool box.  The others were way too big for the delicate projects I worked on.  Although I do admit, I once used the biggest screw driver to beat the living crap out of an 8-track player.  I didn’t fix it, but I felt better.

There are 2 groups of people that know absolutely nothing about software testing.  First – test tool vendors.  Second – Software Project Managers (but not you Kathy).  Sadly, the first group is keenly aware of the existence of the second group’s lack of testing knowledge/experience and are like the old traveling salesman when peddling their wares.  Tool vendors remind me of Starbuck in “The Rainmaker”!  Well then call me Lizzie!  I’m not buying it!  (if I’ve lost you – go see the play)

If you are an automated tool vendor – keep reading (by the way – you’re gonna hate me by the end of this – stand in line – behind the Agilistas).  If you’re currently involved in any automated testing project – keep reading.  If you think automation is always the answer – ignore the vendors and keep reading.  Bottom line – automated crap=diarrhea!  If your underlying tests are crap, automation just gives you high speed crap.  Its still crap.

Any test automation is expensive – even if you use a free tool.  In fact, I’m willing to bet a super-deluxe, cream-filled, chocolate donut that free tools, in the long run, cost more than the big high-dollar tools to implement and use effectively.  Not that those big tool vendors are off the hook.

Please don’t think that I’m against test automation – nothing is further from the truth.  Many of the tools are great tools – when used properly and in the right circumstances.  If it is the right tool for the job.  Will automated tools reduce test times? – mostly.  Will they improve your testing? – maybe.  Will they result in a higher quality end product? – rarely.

Much depends on what you choose to automate, or how you define automation.  Engrave the following somewhere in your brain where you can refer to it often…”You cannot, and in most cases, SHOULD NOT, automate everything.  It is rarely economically feasible!  Still with me tool vendors?  Everyone else pay attention!

Let’s take a quick look at functional test tools.  If you only use the test automation tool to run positive, “happy path” tests can you really consider yourself “automated”?  Not really.  Not in my opinion anyway.  The problem with most “free” tools is that all they do is record user responses and play them back at high speed.  Errors page or messages can fly by – just like the good pages or confirmation messages.  If you are not paying attention, you may miss them unless the system crashes.  These tools do not evaluate or validate anything.  That part is still up to the tester.  Just because a test script runs end to end, it may not mean the test passed or the application works.

Higher functioning tools allow you to modify the input data, which is a step in the right direction.  You can now enter bad data and see what happens.  Most tools typically still require some type of manual validation of the results.  Unless it is a critical error and the system crashes, most scripts will sail past error messages. If you are inducing errors, you probably want to see the results.  I would.

The really good tools will not only let you input good and bad test data, but will also evaluate the results.  These are usually not the free tools.  Really good tools will validate page loads or data fields, detect error messages, read parts of the screen, validate databases, etc.  All of the things you would typically do during a manual test. 

Many of the automated test projects that I have evaluated as a consultant fall way short of the level of testing that I would expect.  Most of the tools are purchased by someone far outside of the testing arena who have no clue what they have just bought – usually someone in Purchasing.  They listen to all the tool vendor hype – typically from a salesman with absolutely zero testing experience, and purchase the tool without talking to anyone from the test team.  Then like Santa Claus, bring you this nicely wrapped “gift” that ends up being completely useless.  It ends up on a shelf gathering dust.

So you load the new tool, designate one or two people as “automated testers” and record and run your first test.  You feed in all the right data in all the right places and click all the right buttons.  The test passes.  On to feature number 2.  You demonstrate your library of new high-speed tests to a completely ignorant and clueless group of managers who are wicked impressed.  You continue to automate the remaining suite of tests.  You deliver the product ahead of schedule – all as a direct result of your automated testing.  You now declare yourself “automated”.

But are you really?  At this point all you really have is an automated smoke test.  It’s a start but there is a huge amount of testing remaining.  Don’t forget the negative side.  If you use a free tool you may have no other option than to do only positive tests.  That is typically all these tools will support.  They are usually just record and playback tools.  They are of limited value.  But hey, what do you want for free?

Oh – and a quick word on automated development test tools – they are just that.  They are not a replacement for functional or system level UI-driven tests.  They work under the covers.  You will eventually need a tool that will drive your test from the UI – as the users will.  Rely solely on this “under-the-covers” tool and you are at risk for huge problems down the application lifecycle,  Or worse – after delivery.

Lastly, in spite of what the traveling salesmen tell you – automated tools are not a plug-n-play option.  They require specialized knowledge, development time, and maintenance.  All still pretty expensive.  Once the system is up and running and all of the scripts are recorded and errors are fixed, you may begin to see some cost savings.  Maybe!

Do I hear thunder?  Is that rain?  Bang that drum!  Louder!

I Test… Therefore Software Works!

Why did I become a Software Tester?  It was an epiphany really.

Ever since I was a wee lad, I wanted to be a Maine fisherman.  I’m actually from a small island off the coast of Maine.  Well technically it’s not an island anymore since they built a road to it. 

My father, born and raised on the island, would always fill me with romantic stories of growing up on the island and working on the boats.  He enlisted in the Air Force after graduating from high school.  He married my Mom and a year later I was born.  During his 20-something year career, we moved around a lot.  Typically far from Maine.  We would go home every now and then for “Old Home Week” – the island’s week-long Fourth of July celebration.  I followed in my Dad’s footsteps and enlisted in the Air Force right after high school.  My dream, after I retired from the Air Force, was to return home to Maine and be a fisherman.  Deep sea fishing. 

For our “honeymoon” I took my new blushing bride home to Maine for the Forth.  I remember it like it was yesterday.  As we approached the “island” from the mainland, the fog was just lifting, the setting sun glowed orange off of the old white houses on the hillside (shacks really).  It was picture perfect.  Like a picture in a calendar.  The following morning we would be on the fishing boat – I was down right giddy!

We got up really early the next morning.  Even though it was July, the air was crisp and cool.  Steam was gently rising off of my freshly brewed cup of coffee.  There was a thick blanket of fog on the water.  You could hear seagulls somewhere in the distance.  It was perfect!  I boarded the boat ready to experience what my future held.

Reality set in about the time the coffee ran out.  The fog lifted after about an hour and it was hot and muggy.  The lifting fog also released the stench of dead fish.  The seagulls were starting to get annoying.  They smelled the dead herring and were circling the boat looking for food.  We were dodging seagull poop.  The tide was out adding a special aroma to the mix.  We were gagging on diesel smoke.  I spent the first hour or so helping my Dad bait the novice fishermen’s hooks with dead, rotting, slimy herring.  My clothes reeked.  And there was still 4 hours to go! 

The only thing we caught after 5 of the longest hours of my life was a squid.  My new wife caught it.  She didn’t really catch it.  It had the misfortune of swimming by just as she was reeling in her line.  She just kinda snagged its head.  The girls all screamed.  No one wanted to touch it.  So my Dad (who I swear was the model for the Gorton’s fisherman) volunteered to unhook it.  It wrapped its tentacles around his arm and used its sticky sucker-thingys to hang on to him for dear life.  He eventually peeled it off and returned it to the bay.  Somewhat traumatized by the ordeal, and with the exception of having once had a huge fishing hook in its head, the squid was pretty much unharmed.

After the squid ordeal, I looked up as we slowly sailed past the island.  I found myself jealous of the lucky islanders in their cozy cottages up on the hill, eating a nice hot breakfast. I longed to be back in my nice soft bed, snuggled up against my new wife, or sprawled out on the couch watching television with the smell of sizzling bacon wafting through the house.  I have no clue who would have been cooking it, but that’s not important right now.

I couldn’t wait to get off that stupid boat!  Needless to say, that was my last fishing trip. 

A week later we returned to Colorado.  On the flight home I pondered my future career aspirations.  Whatever I decided to do was going to be indoors, temperature controlled, and not reek of fish.  I hate fish  I don’t even want an aquarium!  It’s been almost thirty years and I haven’t been fishing since.

So here I am – a Software Tester.  I couldn’t be happier!

I Did It My Way….

Thanks Frank!

I recently decided to take a break from managerial roles and return to life as a simple tester.  I assumed the stress would be less, I’d attend fewer meetings, I could just concentrate on my test assignments and go home at the end of the day and leave work at work.  Yeah right!  The bottom line – I really missed getting my hands dirty and doing the actual testing.  My role as a test manager was much more “conceptual”.  Providing test estimates, writing test plans and strategies, giving my “expert opinion.”  And seemingly endless meetings!  When I wasn’t in a meeting, I was in an office all by myself.  Rarely interacting with anyone.  It killed me.  I absolutely hated it!  I know – I’ll be a consultant!

As a consultant it’s easy to shift gears a little,  I just had to modify my resume and remove all the managerial stuff and replace it with hands-on testing stuff.  Then cast it into the water and see what bites.  I got a lot of bites.  But in spite of my best efforts, I was typically found to be over-qualified.  A bit more tweaking was necessary.  So I reworked my resume again and took a look at some of my responses to interview questions and revised them.  It worked – I’ve been able to land a couple of pure testing roles.  This is usually where the “and he lived happily ever after” line usually comes in.  Not so much.

I found it really hard to take off my test manager’s hat.  I was pretty successful as a test manager.  I’ve written about test management, spoken about test management and taught test management,  I know everything there is to possibly know about test management.  I thought to myself – these people don’t know how lucky they are to get me.  I could assume my “tester” role and at the same time help them improve their testing skills, test planning, defect management etc.  It didn’t quite work out that way.

I learned a few things about myself though.  First, that I can be a little over-bearing.  OK – a lot.  I’ve had to learn to just “shut up and test.”  After all, that’s what they hired me to do.  Not improve their test management.  But it’s hard.  After 20 years of doing this I’ve seen what works and what doesn’t.  Whenever I saw something about to be implemented that had failed for me in the past I spent more time raising the “this ain’t gonna work” alarm rather than helping the team move forward.  I admit it, I was somewhat of a whiner.  That’s OK though – I’d just save up a big “I told you so” to use at the first opportunity.  I rarely got that far.

Second – my opinions were rarely wanted.  Just test!  But you don’t understand – I am after all…me!  People pay money to listen to me (sorta) and I’m giving it to you for free.  What’s wrong with you people?

So what I’ve learned is this.  Just because they don’t do things my way, it doesn’t make them wrong.  They’re just different.  Give ‘em a chance.  If I don’t like something or disagree – discuss it as a team member rather that crashing down on them with my “vast testing knowledge and experience”.  People appreciate listening to ideas rather than having them shoved down their throats by “the consultant.”  Consult – don’t dictate!  Tell them your experience, offer suggestions, be there to help.  I now tell test managers that I may disagree with something (I probably will).  Take it for what it is – an idea.  It’s me.  It’s what I do.  You don’t have to agree with me, but at least give me the opportunity to stick my two-cents worth in.  Let me share my ideas and opinions – in private!  You don’t have to agree with me – just listen.  At the end of the day, we may disagree behind closed doors, but once they make a decision, I’m their biggest cheerleader.

Is it easy?  No.  I still struggle with it – everyday.  Much of it is just my personality.  Most of my previous consulting gigs were aimed at process improvement.  Analyze, and make recommendations to improve.  That’s not my role anymore.  I’ll be honest, maybe it’s time for me to return to a test management role or a problem solving consultant role.  But until then – with the exception of this blog and my quarterly “Last Word…” articles in The European Software Tester (T.E.S.T) magazine – I’ll just “shut up and test.”

I Will Test With Gumbo!

Pay back.

I was at a conference recently and one of my tens of fans approached me in the hallway after one of my highly spirited anti-Agile presentations.  More of a critic than a fan really.  She basically asked me:  “If you hate Agile so much, what can you give us that is better?”  Good point I guess, even though my entire presentation was related to doing things in the true spirit of Agile stressing, of all things, flexibility.  She admitted she had not attended so I cut her some slack.

Her comments stuck with me though – it was time for me to put up or shut up.  This wasn’t the first time I had been challenged on my Anti Agile-ness.  I once had an extensive e-mail/text message conversation with one of the authors and staunch proponents of Agile.  To their credit – they offered their side and listened while I presented mine.  We agreed to disagree and parted on good terms.  I have huge respect for this author! 

But – even though I have always offered an alternative to Agile, I never gave it a catchy name.  Until now.  Ready for it?   Drum roll….   Gumbo!

I thought about this long and hard before I came up with Gumbo.  I wasn’t crazy about the name at first but after some thought, I’m quite satisfied with it. For those of you not familiar with Gumbo let me explain.  Gumbo is very popular in the Southeastern United States – particularly Louisiana.  Gumbo is basically a soup or stew.  Required fare on Mardi Gras Day!  Gumbo chefs in the southeast, amateur or professional, each have their own unique gumbo recipe.  Kinda like chili or Bar-B-Que in Texas.  I’m no exception.  Although I hardly consider myself a chef.

There are some core ingredients to most Gumbo recipes, and then each chef provides their own unique twist.  They may modify the ingredients a bit – usually by adding or substituting something.  Most gumbos have some kind of seafood – usually shrimp – unique to the region.  I adjust mine to the intended diners.  I add what they like.  Subtract what they don’t like.  Or substitute something.  I hate seafood, and my son is allergic to it so I don’t use seafood.  Instead, I substitute chicken and sausage.  Everything else is pretty much traditional.  Not any kind of sausage mind you.  The sausage is unique – when I can get it.  I prefer andouille sausage.  Andouille is a French sausage made with pork, pepper, onion, wine, and seasonings.  It is huge in most Cajun cooking.  In Louisiana, its readily available.  In Colorado, not so much.  Sadly, I’ve yet to find an adequate substitute.

Anyway….I told you that to tell you this.  I apply a similar approach to software testing.  Let’s call it – The Gumbo Process.  There, I named it.

The Gumbo Software Testing Process (patent pending) is much like it’s inspiration.  Whenever I’m on a consulting engagement I typically bring my collection of testing recipes.  Depending on the client, I may use a different recipe or develop an entirely new one.  There are some core practices that I bring with me, and others I may add or subtract depending on what tools are available, tester experience, etc.  No two solutions are exactly the same.  Similar – maybe. Identical – rarely.

That was my initial understanding of Agile.  Flexibility!  Sadly, Agile authors and consultants have destroyed the original intent.  Agile has become a “named process” with very strict rules as to what is and what isn’t Agile.  There are Agile books, conferences, seminars, courses, etc.  I’ve seen Agile evangelists go into companies and thoroughly remove a well working process because it wasn’t “Agile”.  That’s when I jumped off the Agile bandwagon (well that, and it was full!).

So there it is…my alternative to Agile.  Gumbo!  Take it or leave it.  Just remember, I said it first and my brother is an attorney.

Oops!…Now What?

If there is one area that seems to be consistently over looked during the software design process, it’s Error Handling.

A well-designed application will detect and report any error to the user long before it becomes a major issue – like trying to store bad data in the database.  Don’t rely on just reporting system error messages, or the dreaded yellow screen in ASP.NET, to let the user know they messed up.  In fact – never let developers do that!  Good error handling will let the user know, in a friendly manner, that they messed up and will guide them towards an appropriate resolution.  Too many times I’ve seen system error messages that went directly to the UI.  Not only does it completely, and sometimes unnecessarily, terrify the user, it doesn’t tell them what they did wrong or how to resolve it.  OK – it does, but usually in developer-speak.  Something like “AccountNumber is NaN”.  Huh?

Typically, if we don’t tell the developer how to handle and report errors, they will be left to their own devices.  The results: inconsistency – the same type of error is reported differently in different areas of the application.  Omission – errors may be missed (until the database crashes).  Errors are reported in developer-speak or system messages.  I even had an application developed off-shore report an error in Chinese.  The text was red so I assumed I had messed up.

I like to focus on errors and error handling early.  Hopefully in the design/requirements process.  I discreetly remind them that if they ignore me now, they do so at their own peril later on.  I make a note and we move on.  By the way, when you do find the error, try to resist the “I told you so” comment.  Gloat in private.

There are lots of ways to handle and report errors.  Some of them work well.  Others, not so well.  Here are some of the things I typically look for:

 - Consistency:  The same type of error gets the same message no matter where it occurs in the application.  Same error level, text, icon, color, etc.  And the error always appears in the same place (top of the page, next to the data field, etc.)
 
 - Obvious:  Don’t make me look for it.  Make it highly visible.  Don’t make me scroll to find it.  I hate it when an application tells me there is an error but doesn’t show me where.  Or worse, just reloads the page and makes me guess what happened.
 
 - Feedback:  If you can, let the user know not only what they did wrong, but how to fix it.  Exception:  you can be vague on log on pages.  You don’t want to guide a potential hacker to the correct solution.
 
 - Colors:  Different colors mean different things.  Red=danger, yellow=caution, green=good to go, blue =information.  Use a color appropriate to the error.  Don’t tell me the system is about to crash in blue.
 
 - Icons:  Again, like colors, if you use them, use an icon appropriate for the error.  Typical icons I’ve seen: stop signs, caution signs, question marks, check marks.  If you use color as well, follow the color rules above.  If everything worked, don’t show the user a red check mark.
 
 - Avoid flashing things or sounds:  Unless you are about to detonate a nuclear missile.  Then its OK.
 
 - Watch the text:  Don’t be judgmental or long-winded.  Quick, short, and to the point.
 
 
 Best Practices
 
Once again, my apologies to the Context-Driven folks – some Best Practices.  No wait – I’m not apologizing – I stand by my use of “Best Practices”!   Here are some things I’ve seen in the past and really like:
 
 - Combine colors with icons.  For example, icon,  text or background colors fit the error level

 - Use Regular Expressions: Or tell developers to use them.   A RegEx is a great way to detect errors.  Store them in a database table and call them as needed.  Helps to insure consistency.  There are a bunch of regular expressions already written for them on the Internet.  There are also some great RegEx checkers.
 
 - Message Catalog:  Create a table in the database to store all messages reported to the user.  You can also include the message level:  Informational, Caution, Danger, etc.  Don’t forget positive messages such as:  “Record Updated”, “Mail Sent”, etc.  The Message Catalog comes in handy if you ever need to localize your application – trust me!
 
 - Write an Error Handling Guide or Specification:  Hopefully before the first line of code is ever written.
 
 Never over-estimate your user.  They will mess up !  Be one step ahead of them.  Think “Defect Prevention”!

Best Practices (The Context Driven Folks Are Gonna Crucify Me!)

Eh….It wouldn’t be the first time.  I doubt it will be the last.

Let me say up front that I agree with most of what they have to say.  The only part I really have a problem with is the first tenant -” There are no best practices…”   I partially agree.  Is there one single, works in every situation best practice?  Of course not!  Still with me Agile Disciples?

I’ve learned a lot from observing how people do things.  Especially the ones that do things well.  It is something I learned from a very wise, yet crusty, Senior NCO that I used to work for when I was in the  Air Force.  Field Training Detachment (FTD) 910, Hahn Air Base, Germany.  We would get inspection reports all the time from similar units that just completed a major inspection.  90% were rated Satisfactory, 1% got the absolute highest rating – Outstanding (the rest failed).  In any of the reports if a unit did something very well it was highlighted in the final report by the inspection team.  Usually with a good description of what they did to achieve the highest rating – a “best practice”.   We would comb thru these reports and compare what they did to what we were doing.  Then we would try to do as well or better. Of course the Failed reports were valuable too.  We made sure we were not doing the same thing!   We did this for every process or task we had.  The result – we got an Outstanding Rating!  In most cases, we now owned the best practice.  It was by far the most rewarding job I ever had!

There are hundreds, if not thousands of best practices.  For any give problem, issue, or situation there are some people who have put a lot of thought into it.  They found a great way to do it.  They are successful and they should be looked at as an example of how to do something well.   Role models.

The life-cycle of any best practice is short however.  It’s only a best practice until someone makes it better.  Like we did.  Eventually someone probably topped us (they had not before I left).

Will any Best Practice apply 100% – of course not!  But there may be pieces of it that work very well.  Take it as a foundation – a starting point –  and modify it to meet your needs.  I probably carry around dozens of best practices to every consulting engagement I’m on.  I pull out the ones that best fit, then tweak them as needed.  Voila!  A new best practice is born. 

Now ’THAT’s Agile!  OK – bring on the hammer and nails – I’m ready.

A Test Celebrity Worth Seeing And Reading!

If you keep up with my testing rants and raves you know I have little, if any, respect for most of the “Test Celebrities” currently on the book and lecture circuit.  There are a handful that I thoroughly despise.  They’re arrogant, conceited, self-centered, and other adjectives as well!  I’ve met some of them, not all.  I’ve also listened to their presentations – when I could stay awake.  Most have a “My Way Or The Highway” mentality.  You must do as they say or you will be the subject of public ridicule.  I personally experienced it when I dared to question one – politely.   How dare we question them?  Afterall – they wrote a book!  I’m just some no name tester.  They ganged up on me like a pack of starving wolves.  One in particular was especially nasty.  I once witnessed three of them gang up on a conference speaker that they disagreed with.  In the middle of the presentation!

But, I’m happy to say there are a small handful of test celebrities worth listening to and worth reading.  I like them for three reasons.  First, they know what they are talking about and communicate it very well.  Second, they are personable and pleasant to chat with.  Even when they disagree with you, they are respectful.  Lastly, and most importantly, the other group despises them.  For me, that alone is reason enough!   James Whittaker makes that short list.

If you ever get a chance to listen to James Whittaker – take it!  Trust me, not only will you learn something, he is a pleasure to listen to.  I recommend you read all of his books but especially his latest one “Exploratory Testing.”  As far as I’m concerned its pure genius!

Bottom line – James just gets it!