I once had a development manager approach me and advise me that he had just hired a developer that he had known for years. He bragged that in all the years he has worked with this developer, the developer had only created five bugs. I maintained my composure and tried had not to burst out into hysterical laughter. This developer had obviously not met me.
I am a Software Entomologist – bugs are my life! I study them. I know where they live. I will find them. There is no such thing as perfect software and I’ll prove it! In this case I did. As project’s Test Manager, I assigned myself this developer’s features and proceeded to test them. I have to admit, the code was very well written. The bugs were hard to find but I found them – 11 total. But I had to work really hard to find them and it ate up a lot of valuable time in the test schedule. Was it worth it?
I’ve been giving a lot of thought to what we do lately. I tend to read a lot of things from my fellow software testers and from those I like to refer to as “Test Celebrities” (you know, the ones that write the books and speak at the conferences). Its nice to know what they’re up to, what the latest trends are, etc. I have found that lately they seem to spend a lot of time telling us how to find those really elusive, hard to find bugs in an application. One in particular is somewhat obsessed with the subject. I have admit, I blindly followed their advice and have built a reputation as someone who can typically find those elusive bugs. But is that what I should be doing?
I don’t think so. Our job is to make sure the application works as it was designed. That the apllication is functional and usable. That the user is gracefully led back onto the path should they stray – and they will stray. Users, especially in Web applications where there is little or no documentation, sometimes tend to get lost or do things they really shouldn’t do. Those are the things that I need to brainstorm and test for. There is a lot more value to me entering invalid data, submitting empty forms, violating database constraints, and stuff like that than there is to holding down the Alt key while pressing Q on during a leap year. Sure that may break something, but do we really care? What are the odds a typical user (not one like me) would do that. By the way, in the event that you do encounter a user like me – thank them and buy them a cup of coffee. If the odds are pretty good that a user could do it – test it. If the odds are pretty low….maybe later if there is time left in the schedule (because there is always time left over at the end of the schedule right?). Well thought out negative testing adds value. I think the jury is still out on exploratory testing and bug quests. I may be wrong. I’ve been wrong before and I’m sure I’ll be wrong again.
I guess that is why I’m not a huge fan of exploratory testing (as I understand it anyway). I prefer a well structured, methodical, highly repeatable approach. Both positive and negative testing. Is there some value to exploratory testing? Maybe. If I have time, I’ll go on a bug quest. I will find bugs. Developers will despise me, but hopefully most will be low priority bugs and they will get fixed eventually. But since I rarely have the time in any test cycle to do the testing that I need to do, I don’t really see much exploratory testing happening.
Can we find all the bugs in a software application? Given sufficient time and resources, I think we can get very close. But it would take so long to test and as a result be so expensive that it wouldn’t be economically feasible. But I do love a dare.