Is Automation Always The Answer? Not So Fast…

When I was a teenager – many years ago – Santa bought me a new set of tools.  I had my license and an old used Toyota.  Santa apparently thought I needed tools.  It was a really nice auto mechanics tool set, complete with a really nice tool box.  Not to seem ungrateful, but I knew absolutely nothing about auto mechanics.  40-some years later, I still don’t.  I fixed electronic stuff like radios, TVs, and stereos – still do.  Mostly computers these days.  Even if I had the first clue about auto mechanics – I drove a Toyota.  The tools were standard, I needed metric.  With the exception of the set of screw drivers, I couldn’t do much with it.  In fact, I could only really use 3 of the 12 screw drivers in the tool box.  The others were way too big for the delicate projects I worked on.  Although I do admit, I once used the biggest screw driver to beat the living crap out of an 8-track player.  I didn’t fix it, but I felt better.

There are 2 groups of people that know absolutely nothing about software testing.  First – test tool vendors.  Second – Software Project Managers (but not you Kathy).  Sadly, the first group is keenly aware of the existence of the second group’s lack of testing knowledge/experience and are like the old traveling salesman when peddling their wares.  Tool vendors remind me of Starbuck in “The Rainmaker”!  Well then call me Lizzie!  I’m not buying it!  (if I’ve lost you – go see the play)

If you are an automated tool vendor – keep reading (by the way – you’re gonna hate me by the end of this – stand in line – behind the Agilistas).  If you’re currently involved in any automated testing project – keep reading.  If you think automation is always the answer – ignore the vendors and keep reading.  Bottom line – automated crap=diarrhea!  If your underlying tests are crap, automation just gives you high speed crap.  Its still crap.

Any test automation is expensive – even if you use a free tool.  In fact, I’m willing to bet a super-deluxe, cream-filled, chocolate donut that free tools, in the long run, cost more than the big high-dollar tools to implement and use effectively.  Not that those big tool vendors are off the hook.

Please don’t think that I’m against test automation – nothing is further from the truth.  Many of the tools are great tools – when used properly and in the right circumstances.  If it is the right tool for the job.  Will automated tools reduce test times? – mostly.  Will they improve your testing? – maybe.  Will they result in a higher quality end product? – rarely.

Much depends on what you choose to automate, or how you define automation.  Engrave the following somewhere in your brain where you can refer to it often…”You cannot, and in most cases, SHOULD NOT, automate everything.  It is rarely economically feasible!  Still with me tool vendors?  Everyone else pay attention!

Let’s take a quick look at functional test tools.  If you only use the test automation tool to run positive, “happy path” tests can you really consider yourself “automated”?  Not really.  Not in my opinion anyway.  The problem with most “free” tools is that all they do is record user responses and play them back at high speed.  Errors page or messages can fly by – just like the good pages or confirmation messages.  If you are not paying attention, you may miss them unless the system crashes.  These tools do not evaluate or validate anything.  That part is still up to the tester.  Just because a test script runs end to end, it may not mean the test passed or the application works.

Higher functioning tools allow you to modify the input data, which is a step in the right direction.  You can now enter bad data and see what happens.  Most tools typically still require some type of manual validation of the results.  Unless it is a critical error and the system crashes, most scripts will sail past error messages. If you are inducing errors, you probably want to see the results.  I would.

The really good tools will not only let you input good and bad test data, but will also evaluate the results.  These are usually not the free tools.  Really good tools will validate page loads or data fields, detect error messages, read parts of the screen, validate databases, etc.  All of the things you would typically do during a manual test. 

Many of the automated test projects that I have evaluated as a consultant fall way short of the level of testing that I would expect.  Most of the tools are purchased by someone far outside of the testing arena who have no clue what they have just bought – usually someone in Purchasing.  They listen to all the tool vendor hype – typically from a salesman with absolutely zero testing experience, and purchase the tool without talking to anyone from the test team.  Then like Santa Claus, bring you this nicely wrapped “gift” that ends up being completely useless.  It ends up on a shelf gathering dust.

So you load the new tool, designate one or two people as “automated testers” and record and run your first test.  You feed in all the right data in all the right places and click all the right buttons.  The test passes.  On to feature number 2.  You demonstrate your library of new high-speed tests to a completely ignorant and clueless group of managers who are wicked impressed.  You continue to automate the remaining suite of tests.  You deliver the product ahead of schedule – all as a direct result of your automated testing.  You now declare yourself “automated”.

But are you really?  At this point all you really have is an automated smoke test.  It’s a start but there is a huge amount of testing remaining.  Don’t forget the negative side.  If you use a free tool you may have no other option than to do only positive tests.  That is typically all these tools will support.  They are usually just record and playback tools.  They are of limited value.  But hey, what do you want for free?

Oh – and a quick word on automated development test tools – they are just that.  They are not a replacement for functional or system level UI-driven tests.  They work under the covers.  You will eventually need a tool that will drive your test from the UI – as the users will.  Rely solely on this “under-the-covers” tool and you are at risk for huge problems down the application lifecycle,  Or worse – after delivery.

Lastly, in spite of what the traveling salesmen tell you – automated tools are not a plug-n-play option.  They require specialized knowledge, development time, and maintenance.  All still pretty expensive.  Once the system is up and running and all of the scripts are recorded and errors are fixed, you may begin to see some cost savings.  Maybe!

Do I hear thunder?  Is that rain?  Bang that drum!  Louder!

I Did It My Way….

Thanks Frank!

I recently decided to take a break from managerial roles and return to life as a simple tester.  I assumed the stress would be less, I’d attend fewer meetings, I could just concentrate on my test assignments and go home at the end of the day and leave work at work.  Yeah right!  The bottom line – I really missed getting my hands dirty and doing the actual testing.  My role as a test manager was much more “conceptual”.  Providing test estimates, writing test plans and strategies, giving my “expert opinion.”  And seemingly endless meetings!  When I wasn’t in a meeting, I was in an office all by myself.  Rarely interacting with anyone.  It killed me.  I absolutely hated it!  I know – I’ll be a consultant!

As a consultant it’s easy to shift gears a little,  I just had to modify my resume and remove all the managerial stuff and replace it with hands-on testing stuff.  Then cast it into the water and see what bites.  I got a lot of bites.  But in spite of my best efforts, I was typically found to be over-qualified.  A bit more tweaking was necessary.  So I reworked my resume again and took a look at some of my responses to interview questions and revised them.  It worked – I’ve been able to land a couple of pure testing roles.  This is usually where the “and he lived happily ever after” line usually comes in.  Not so much.

I found it really hard to take off my test manager’s hat.  I was pretty successful as a test manager.  I’ve written about test management, spoken about test management and taught test management,  I know everything there is to possibly know about test management.  I thought to myself – these people don’t know how lucky they are to get me.  I could assume my “tester” role and at the same time help them improve their testing skills, test planning, defect management etc.  It didn’t quite work out that way.

I learned a few things about myself though.  First, that I can be a little over-bearing.  OK – a lot.  I’ve had to learn to just “shut up and test.”  After all, that’s what they hired me to do.  Not improve their test management.  But it’s hard.  After 20 years of doing this I’ve seen what works and what doesn’t.  Whenever I saw something about to be implemented that had failed for me in the past I spent more time raising the “this ain’t gonna work” alarm rather than helping the team move forward.  I admit it, I was somewhat of a whiner.  That’s OK though – I’d just save up a big “I told you so” to use at the first opportunity.  I rarely got that far.

Second – my opinions were rarely wanted.  Just test!  But you don’t understand – I am after all…me!  People pay money to listen to me (sorta) and I’m giving it to you for free.  What’s wrong with you people?

So what I’ve learned is this.  Just because they don’t do things my way, it doesn’t make them wrong.  They’re just different.  Give ‘em a chance.  If I don’t like something or disagree – discuss it as a team member rather that crashing down on them with my “vast testing knowledge and experience”.  People appreciate listening to ideas rather than having them shoved down their throats by “the consultant.”  Consult – don’t dictate!  Tell them your experience, offer suggestions, be there to help.  I now tell test managers that I may disagree with something (I probably will).  Take it for what it is – an idea.  It’s me.  It’s what I do.  You don’t have to agree with me, but at least give me the opportunity to stick my two-cents worth in.  Let me share my ideas and opinions – in private!  You don’t have to agree with me – just listen.  At the end of the day, we may disagree behind closed doors, but once they make a decision, I’m their biggest cheerleader.

Is it easy?  No.  I still struggle with it – everyday.  Much of it is just my personality.  Most of my previous consulting gigs were aimed at process improvement.  Analyze, and make recommendations to improve.  That’s not my role anymore.  I’ll be honest, maybe it’s time for me to return to a test management role or a problem solving consultant role.  But until then – with the exception of this blog and my quarterly “Last Word…” articles in The European Software Tester (T.E.S.T) magazine – I’ll just “shut up and test.”

I Will Test With Gumbo!

Pay back.

I was at a conference recently and one of my tens of fans approached me in the hallway after one of my highly spirited anti-Agile presentations.  More of a critic than a fan really.  She basically asked me:  “If you hate Agile so much, what can you give us that is better?”  Good point I guess, even though my entire presentation was related to doing things in the true spirit of Agile stressing, of all things, flexibility.  She admitted she had not attended so I cut her some slack.

Her comments stuck with me though – it was time for me to put up or shut up.  This wasn’t the first time I had been challenged on my Anti Agile-ness.  I once had an extensive e-mail/text message conversation with one of the authors and staunch proponents of Agile.  To their credit – they offered their side and listened while I presented mine.  We agreed to disagree and parted on good terms.  I have huge respect for this author! 

But – even though I have always offered an alternative to Agile, I never gave it a catchy name.  Until now.  Ready for it?   Drum roll….   Gumbo!

I thought about this long and hard before I came up with Gumbo.  I wasn’t crazy about the name at first but after some thought, I’m quite satisfied with it. For those of you not familiar with Gumbo let me explain.  Gumbo is very popular in the Southeastern United States – particularly Louisiana.  Gumbo is basically a soup or stew.  Required fare on Mardi Gras Day!  Gumbo chefs in the southeast, amateur or professional, each have their own unique gumbo recipe.  Kinda like chili or Bar-B-Que in Texas.  I’m no exception.  Although I hardly consider myself a chef.

There are some core ingredients to most Gumbo recipes, and then each chef provides their own unique twist.  They may modify the ingredients a bit – usually by adding or substituting something.  Most gumbos have some kind of seafood – usually shrimp – unique to the region.  I adjust mine to the intended diners.  I add what they like.  Subtract what they don’t like.  Or substitute something.  I hate seafood, and my son is allergic to it so I don’t use seafood.  Instead, I substitute chicken and sausage.  Everything else is pretty much traditional.  Not any kind of sausage mind you.  The sausage is unique – when I can get it.  I prefer andouille sausage.  Andouille is a French sausage made with pork, pepper, onion, wine, and seasonings.  It is huge in most Cajun cooking.  In Louisiana, its readily available.  In Colorado, not so much.  Sadly, I’ve yet to find an adequate substitute.

Anyway….I told you that to tell you this.  I apply a similar approach to software testing.  Let’s call it – The Gumbo Process.  There, I named it.

The Gumbo Software Testing Process (patent pending) is much like it’s inspiration.  Whenever I’m on a consulting engagement I typically bring my collection of testing recipes.  Depending on the client, I may use a different recipe or develop an entirely new one.  There are some core practices that I bring with me, and others I may add or subtract depending on what tools are available, tester experience, etc.  No two solutions are exactly the same.  Similar – maybe. Identical – rarely.

That was my initial understanding of Agile.  Flexibility!  Sadly, Agile authors and consultants have destroyed the original intent.  Agile has become a “named process” with very strict rules as to what is and what isn’t Agile.  There are Agile books, conferences, seminars, courses, etc.  I’ve seen Agile evangelists go into companies and thoroughly remove a well working process because it wasn’t “Agile”.  That’s when I jumped off the Agile bandwagon (well that, and it was full!).

So there it is…my alternative to Agile.  Gumbo!  Take it or leave it.  Just remember, I said it first and my brother is an attorney.

Best Practices (The Context Driven Folks Are Gonna Crucify Me!)

Eh….It wouldn’t be the first time.  I doubt it will be the last.

Let me say up front that I agree with most of what they have to say.  The only part I really have a problem with is the first tenant -” There are no best practices…”   I partially agree.  Is there one single, works in every situation best practice?  Of course not!  Still with me Agile Disciples?

I’ve learned a lot from observing how people do things.  Especially the ones that do things well.  It is something I learned from a very wise, yet crusty, Senior NCO that I used to work for when I was in the  Air Force.  Field Training Detachment (FTD) 910, Hahn Air Base, Germany.  We would get inspection reports all the time from similar units that just completed a major inspection.  90% were rated Satisfactory, 1% got the absolute highest rating – Outstanding (the rest failed).  In any of the reports if a unit did something very well it was highlighted in the final report by the inspection team.  Usually with a good description of what they did to achieve the highest rating – a “best practice”.   We would comb thru these reports and compare what they did to what we were doing.  Then we would try to do as well or better. Of course the Failed reports were valuable too.  We made sure we were not doing the same thing!   We did this for every process or task we had.  The result – we got an Outstanding Rating!  In most cases, we now owned the best practice.  It was by far the most rewarding job I ever had!

There are hundreds, if not thousands of best practices.  For any give problem, issue, or situation there are some people who have put a lot of thought into it.  They found a great way to do it.  They are successful and they should be looked at as an example of how to do something well.   Role models.

The life-cycle of any best practice is short however.  It’s only a best practice until someone makes it better.  Like we did.  Eventually someone probably topped us (they had not before I left).

Will any Best Practice apply 100% – of course not!  But there may be pieces of it that work very well.  Take it as a foundation – a starting point –  and modify it to meet your needs.  I probably carry around dozens of best practices to every consulting engagement I’m on.  I pull out the ones that best fit, then tweak them as needed.  Voila!  A new best practice is born. 

Now ’THAT’s Agile!  OK – bring on the hammer and nails – I’m ready.

Herding Cats

Managing defects is kinda like herding cats.  It’s nearly impossible.  Defects, like cats, can’t really be managed.  To be honest, it’s more the process of managing defect resolution than the defects themselves.  As a self-proclaimed software entomologist, I have seen defect management processes that range from absolutely nothing to totally burdensome and unuseable.  Most are so detailed that they seem to severely micro-manage and tie the hands of most QA and development teams. Looking at their process diagrams reminds me of looking at a plate of spaghetti and meatballs.  They make me hungry!

Managing defects isn’t rocket science.  I like to keep my defect management process fairly simple and straight forward.  Simple being the key word!  I like as few states (or statuses) and transitions as possible (developers tend to get easily confused).  You find the bug.  You fix the bug.  You retest the bug.  OK, maybe not that simple.  But close.

In my simple plan, every defect is defined by 2 basic characteristics:  State and Owner.

State:  Determines where a given defect currently resides in the overall defect lifecycle.  Typical states may include:
  - New
  - Assigned
  - Open
  - Resolved
  - Migrated
  - Deferred
  - Closed
 
Owner:  For every given state, a defect is assigned an Owner. The Owner is the person responsible for reviewing, assessing, estimating level of effort to resolve it, and in most cases actually completing the resolution.  Is the Owner always a developer?  Not neccisarily.  It could be a Product Manager, a Business Analyst, or even a tester!  Your list of Owners will typically depend on your organizational structure.  Some examples:
  - Project Management
  - Triage Team
  - Test/QA
  - Development
  - SCM
  
Based on the assessment by the Owner, the defect may move, or Transition, to another state.  Transitions are typically defined as action verbs. 

Transition:  Those actions that are available to a defect owner to move the defect from one state to another.  Depending on a defect’s current state, the available transitions my be limited.  I usually allow the projet manager to override the workflow otherwise I typically restrict a user’s actions.  Transaction may include:
  - Assign
  - Open
  - Close
  - Defer
  - Pass
  - Fail
  - Migrate
  
A final word or two, or three, or more.  Most defect management tools will support customizing the tool to fit your process.  Others – not so much. The not-so-much tools should be kicked to the curb.  Always adapt the tool to fit your process.  Never adapt your process to fit the tool! 

If you want to learn more or have any questions, please feel free to contact me at dwhalen14@gmail.com.  I even have a really spiffy SOT (State-Owner-Transition) diagram that I can share with you.

Rollin’, rollin’, rollin’.  Keep those kitties rollin’  Yee Haw!!!

Making “Nice”

Since I have recently been accused of not being “nice” I figured it was time to write something “positive”. As I have stated here too many times to count – I’m no fan of “A”. There are parts I really like. Other parts – not so much. My position is, and always has been, that software testing is not something we can apply a standard (or for the Context Driven school) a “best practice” to. This includes A. In the early days, that is what A was all about – flexibility! Sadly, those days are over. But back on track. Many A practices use Test Driven Development (TDD) as one of the key components. I whole-heartedly agree!

When I was in the Air Force, I spent a number of years in Technical Training as an Instructor, Instructional Designer/Developer and Curriculum/Course Supervisor. By far my favorite position was Instructional Designer/Developer. As an Instructional Designer/Developers we analyzed the training needs of active duty U.S. Air Force, Air Force Reserve, Air National Guard, and foreign military services on complex F-16 avionics systems.We then built the courses, both initial skills and advanced, to support all the various customers. Back then we used the Instructional Systems Design (ISD) process to create the curriculum.

A major piece of the ISD process is writing objectives and tests. We would write an objective to define an expected training out come and then write the test or measurement to validate that the student successfully met the objective. All other training materials or “content” – lectures, workbooks, student texts, practical exercises, etc., were then written to prepare the students to be able to “pass the test”. The tests may be written tests or practical “hands-on” tests of a given process – like testing a Flight Control Computer. Each course had hundreds of objectives and individual test items. All in all it was a great process. Universal? No. We sometimes had to tweak a few things here and there – but it worked! It was a standard, but allowed flexibility. Like A, many tried to make the rules extremely rigid. It rarely worked. They may still use the process today but I retired from the Air Force 10 years ago so I don’t know.

OK – I told you that to tell you this…

When I was first thrown into Software Test Development (yes, thrown) I quickly found that there is little to no training available. So I applied what I already knew. I had my team write test objectives and then write the test case or test script needed to validate it. The only piece we had no direct control over was the development (content). After a while we learned that if we gave the developers the test objectives and test cases before the code was written we were much more successful. The best part – we could review, verify, and approve the objectives and tests as a team (customers, managers, developers, etc.) before the first line of code was written. We found literally hundreds of areas where blocks of code or tests could be either reused multiple times or simply modified rather than written from scratch. We ended up writing libraries of code blocks and test cases that we could call or reuse as needed. We were doing TDD and didn’t even know it!

The result – the project was delivered on time, within budget, and virtually error free. More importantly – the customer loved it!

Now – we didn’t have 2 week “sprints” or a lot of the other A stuff. We actually did 3 “iterations”. No pigs or chickens – or whatever they’re called. No burn-down charts. None of that stuff. We did have daily 15 minute stand-ups – another practice I love and use universally. We were highly collaborative. It was by far the best project I ever worked on! Were we an “A” team? I think so, but the A purists will tell you no!

Was there a down side? Yep. We did daily builds and ran a daily smoke test. If the smoke test failed, the developer that broke the build bought donuts the next day. If it passed, QA bought the donuts. I gained 20 pounds!

The Angry Tester

I just got finished reading through the blog of one of the “Test Celebrities”.   You know the ones I’m talking about – they write the books, speak at all the conferences, etc.   I have to admit, I don’t like many of them.  I went to a conference once and watched three of them gang up on a poor speaker.  They all share one thing in common – believe as they tell you or else.  How dare anyone disagree with them?  Well I do, often, and I’m saying it here!

My experience also hit home – I never want to be one of them!  I’ll give you my 2-cents worth here, but it is just that….my opinion.  Take it or leave it.  It really doesn’t matter to me.  Please disagree with me – I encourage it!  I’m not the expert in software testing, nor have I or will I, ever claim to be.  Unlike my “colleagues” I would never publically humiliate a speaker.

Anyway – I digress.   This “Test Celebrity” is very angry.  There were two particular hot strings to follow.  The first is an ongoing feud between himself and another celebrity.  The second – someone had the audacity to question the expert.  How dare they!  The really sad part is that they seem to have followers who are just as angry, if not more so.

I disagreed with one of them once.  I was following a software testing mail list and made a comment (nothing  derogatory) about a task they were giving to the members of the list.  Basically they presented a test scenario and asked the members to write in describing how they would test it.   I had seen this tactic before.  They would typically provide a nasty response back to the group if anyone dared to think differently.  I wrote back simply saying that it might be a helpful learning experience if the “celebrity” told the group how he would handle it using their process.  The next day one of  the other “celebrities” added a post to the group publically ridiculing ME! Again, I’m open to disagreement and I have no problem if you disapprove of me or my ideas in a public forum.  But this was just downright nasty!  Something that would have best been delivered in a private email and not a public post.  He was also one of the culprits that ridiculed the conference speaker.  Needless to say, I have absolutely no respect for him or his ideas.

So who do I like?  Very few.  I can count them on one hand.  I read their stuff (books, articles, etc.) all the time and I’ve implemented some of their ideas.  They are not only entertaining – they know their stuff.  I try to never miss a chance to see them.  Three speakers to never miss:  Randy Rice, James Whitaker, and Lee Copeland.  The “always miss” list has 4 members, but I’ll keep that one to myself.  I won’t seek them out, but if they are at a conference I’m attending I’ll stop by for a visit.  Payback is a b%^&*!    GAME ON!   )

I Hate Agile!

OK…before you start the hate mail…maybe hate is too strong a word.  Actually it is not the Agile (or Scrum, or XP) process that I hate, but the Agile practiotioners are really starting to get to me. 

I just finished reading about how one practitioner (I’m not going to name anyone specific….you know who you are) is upset that some newly converted groups tend to fall off the Agile wagon, back into old habits.  He chose of course to publically ridicule them as a group (thankfully, no specific names).  As with most of these “born-again” Agile cultists, if you don’t strictly adhear to their rules you are no longer worthy of their support, and are subject to public condemnation.  Agile has become a religious cult and the practitioners have become like early American religious fanatics (the ones kicked out of England for their fanatic beliefs).  “Believe as we do or we shall excommunicate you from the flock!”   Well then point me to the pillory – I confess – I’m a non-believer!

I wish there were one single, out-of-the-box development process that would work universally for everyone.  There isn’t!  There never will be!  Personally, I like to follow what I call the “Gumbo” process.  A little bit of this, a little bit of that, add what works, subtract what doesn’t – heat and stir and Voila! – Software (or as Eddie Izzard would say: “Hootcha, hootcha, hootcha….software). 

Don’t get me wrong – Agile has some really good ideas, but so does Iterative, RUP, and believe it or not – Waterfall!  That’s right – I said Waterfall – rack me!  Hand me my scarlett letter – I will wear it proudly!