The Software Testing Dichotomy

If you are ramping-up a testing effort for your group, You will eventually arrive at the question, “How much testing is the right amount?” Ultimately, your goal is to iron-out all of the bugs. If you are serious about it, you will want to dedicate enough resources to accomplish this, but you don’t exactly want to write a blank check or wait forever. Hence the question, “How much?”

First, take a step backwards and think about the reasons that you want to eliminate all bugs. Your list is probably like this: 1. Bugs are bad for my/our group’s image. A bad reputation is hard to repair. 2. It costs much less to fix a bug before distribution rather than, afterwards.

If you care about both of these reasons, then you will want to test an app until all of the bugs are worked-out. That is nice and all, but how will you know when the bugs are worked out? Perhaps you believe that one of these will prove it:

  • Your testing has found some bugs (proving that it is effective) but now, that has stopped. This means that the bugs must all be gone.
  • You have poured testing all over your app, up to the ratio of: 1 hour testing for every hour of development. Science has determined that this is the optimal ratio. Therefore, you are all-set.
  • You have gone “fully NASA” and have code-reviewed EVERY single line of code in your app. No bug can withstand our ultimate level of scrutiny. Not even if the bug was made by Chuck Norris.

The reality is this: It is only theoretically possible to find all of the bugs in a program. The less that a program does, the greater your chance of achieving this. In a program with many features, the cost of testing it (enough to reveal/eliminate all bugs) is likely to be cost-prohibitive.

From an economic standpoint, there is a diminishing return on investment, after a certain point. To optimize your investment, you need to cut it off at a reasonable point. In contrast, this means that your users will inevitably find some bugs. You need to accept this reality and determine reasonable responses to protect your reputation.

Testing ROI graph

Some Comp-Sci folk-heroes have even coined a few “laws” about testing, to explain or document this situation. These might seem like jokes, but there is a lot of truth in them:

  • Lubarsky’s Law of Cybernetic Entomology:There’s always one more bug
  • Law of Darwinism of Business Rules:If you produce a program that is goof-proof, the world will produce a better goof
  • General Laws of Computer ProgrammingVIII – Any non-trivial program contains at least one bug

    IX – Undetectable errors are infinite in variety, in contrast to detectable errors, which by definition are limited.

The Answer

I must confess that there is no globally correct answer to this formula. This is why management gets the big bucks. They make decisions about how much tolerance for bugs, is acceptable. The boss picks some criteria for “done” and a back-up plan to handle the bugs that manage to escape. A decision like this might seem arbitrary, but it is just as accurate or scientific as any other informed-decision. It is a gamble. It always will be (to some extent).

There is one exception: The boss can produce an invalid answer if the decision, is to accept nothing-less-than that-which-is-impossible. Namely, “we will only accept bug-free code” or “zero tolerance for bugs”. If you get this kind of requirement from the boss, then he is being irrational.  Please send him (or her) to read this article, and/or others like it.

If you look at the graph (above), when does the bug count get to zero? Never. That is correct. Even spending money up to infinity, will only get you really close to zero-bugs. Good managers don’t make bad investments like that.

Perhaps I should have started this article with a grandiose title like “Welcome to reality land, where perfection is only a theory. Do not expect to wake up, because this is no-longer a dream”. This would sound too snarky, I suppose, and that is not really my point. My actual point is this: Since perfection is impossible, the boss must not ask the developers to perform the impossible. Doing so, is really just passing the buck to (perhaps) unsuspecting developers. If you are the boss, then let the buck stop with you. Live in the real world. Set proper expectations.

There is a big difference between an arbitrary decision and an informed decision. You don’t need to be an elitist to pick the informed one, but being informed, before you make decisions, may lead to elitism.


About Tim Golisch

I'm a geek. I do geeky things.
This entry was posted in Testing and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s