Home » QA Process » Is Automation Always The Answer? Not So Fast…

Is Automation Always The Answer? Not So Fast…

Start here


When I was a teenager – many years ago – Santa bought me a new set of tools.  I had my license and an old used Toyota.  Santa apparently thought I needed tools.  It was a really nice auto mechanics tool set, complete with a really nice tool box.  Not to seem ungrateful, but I knew absolutely nothing about auto mechanics.  40-some years later, I still don’t.  I fixed electronic stuff like radios, TVs, and stereos – still do.  Mostly computers these days.  Even if I had the first clue about auto mechanics – I drove a Toyota.  The tools were standard, I needed metric.  With the exception of the set of screw drivers, I couldn’t do much with it.  In fact, I could only really use 3 of the 12 screw drivers in the tool box.  The others were way too big for the delicate projects I worked on.  Although I do admit, I once used the biggest screw driver to beat the living crap out of an 8-track player.  I didn’t fix it, but I felt better.

There are 2 groups of people that know absolutely nothing about software testing.  First – test tool vendors.  Second – Software Project Managers (but not you Kathy).  Sadly, the first group is keenly aware of the existence of the second group’s lack of testing knowledge/experience and are like the old traveling salesman when peddling their wares.  Tool vendors remind me of Starbuck in “The Rainmaker”!  Well then call me Lizzie!  I’m not buying it!  (if I’ve lost you – go see the play)

If you are an automated tool vendor – keep reading (by the way – you’re gonna hate me by the end of this – stand in line – behind the Agilistas).  If you’re currently involved in any automated testing project – keep reading.  If you think automation is always the answer – ignore the vendors and keep reading.  Bottom line – automated crap=diarrhea!  If your underlying tests are crap, automation just gives you high speed crap.  Its still crap.

Any test automation is expensive – even if you use a free tool.  In fact, I’m willing to bet a super-deluxe, cream-filled, chocolate donut that free tools, in the long run, cost more than the big high-dollar tools to implement and use effectively.  Not that those big tool vendors are off the hook.

Please don’t think that I’m against test automation – nothing is further from the truth.  Many of the tools are great tools – when used properly and in the right circumstances.  If it is the right tool for the job.  Will automated tools reduce test times? – mostly.  Will they improve your testing? – maybe.  Will they result in a higher quality end product? – rarely.

Much depends on what you choose to automate, or how you define automation.  Engrave the following somewhere in your brain where you can refer to it often…”You cannot, and in most cases, SHOULD NOT, automate everything.  It is rarely economically feasible!  Still with me tool vendors?  Everyone else pay attention!

Let’s take a quick look at functional test tools.  If you only use the test automation tool to run positive, “happy path” tests can you really consider yourself “automated”?  Not really.  Not in my opinion anyway.  The problem with most “free” tools is that all they do is record user responses and play them back at high speed.  Errors page or messages can fly by – just like the good pages or confirmation messages.  If you are not paying attention, you may miss them unless the system crashes.  These tools do not evaluate or validate anything.  That part is still up to the tester.  Just because a test script runs end to end, it may not mean the test passed or the application works.

Higher functioning tools allow you to modify the input data, which is a step in the right direction.  You can now enter bad data and see what happens.  Most tools typically still require some type of manual validation of the results.  Unless it is a critical error and the system crashes, most scripts will sail past error messages. If you are inducing errors, you probably want to see the results.  I would.

The really good tools will not only let you input good and bad test data, but will also evaluate the results.  These are usually not the free tools.  Really good tools will validate page loads or data fields, detect error messages, read parts of the screen, validate databases, etc.  All of the things you would typically do during a manual test. 

Many of the automated test projects that I have evaluated as a consultant fall way short of the level of testing that I would expect.  Most of the tools are purchased by someone far outside of the testing arena who have no clue what they have just bought – usually someone in Purchasing.  They listen to all the tool vendor hype – typically from a salesman with absolutely zero testing experience, and purchase the tool without talking to anyone from the test team.  Then like Santa Claus, bring you this nicely wrapped “gift” that ends up being completely useless.  It ends up on a shelf gathering dust.

So you load the new tool, designate one or two people as “automated testers” and record and run your first test.  You feed in all the right data in all the right places and click all the right buttons.  The test passes.  On to feature number 2.  You demonstrate your library of new high-speed tests to a completely ignorant and clueless group of managers who are wicked impressed.  You continue to automate the remaining suite of tests.  You deliver the product ahead of schedule – all as a direct result of your automated testing.  You now declare yourself “automated”.

But are you really?  At this point all you really have is an automated smoke test.  It’s a start but there is a huge amount of testing remaining.  Don’t forget the negative side.  If you use a free tool you may have no other option than to do only positive tests.  That is typically all these tools will support.  They are usually just record and playback tools.  They are of limited value.  But hey, what do you want for free?

Oh – and a quick word on automated development test tools – they are just that.  They are not a replacement for functional or system level UI-driven tests.  They work under the covers.  You will eventually need a tool that will drive your test from the UI – as the users will.  Rely solely on this “under-the-covers” tool and you are at risk for huge problems down the application lifecycle,  Or worse – after delivery.

Lastly, in spite of what the traveling salesmen tell you – automated tools are not a plug-n-play option.  They require specialized knowledge, development time, and maintenance.  All still pretty expensive.  Once the system is up and running and all of the scripts are recorded and errors are fixed, you may begin to see some cost savings.  Maybe!

Do I hear thunder?  Is that rain?  Bang that drum!  Louder!



  1. Joris says:

    Great article! Summarizes some of the traps of automation nicely. Tools cannot evaluate the result of a test properly or comprehensively. That’s a great argument against automation.

    There is a good presentation by Goranka Bjedov on how to use automation in an intelligent way. It can be found here: http://video.google.com/videoplay?docid=-6891978643577501895#
    Maybe you will like it.

  2. Dave Whalen says:

    Thanks Joris – Great link!

  3. Nice Article. Interesting one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: