I was going to try to coin a term here: Testing Maturity Model but then I Bing’d it and found that it already exists(http://en.wikipedia.org/wiki/Testing_Maturity_Model). My concept is a little different from the “official” version, (but not completely different). What can I say, I like my model better. (Big surprise, right?) So, I’d like to go ahead with my impression of what a Testing Maturity Model should be, and how it would apply to your (or my) organization.
I am a pretty big fan of CMM (Capability Maturity Model from Carnegie Melon). It does a great job of describing ways and paths for growing from your current SW dev processes up to better, more mature processes. When you look at CMM, you can recognize that it shows a healthy path for progress and improvement. It is a nice road-map that takes the guess-work out of how to do it. It might also answer frustrating questions, such as “Golly, we have been trying really hard to improve but it isn’t working very well. Why not?!” Usually, it is because you are missing some critical, intermediate steps.
As your dev process grows and matures, it is natural to direct some of your focus to the topic of testing. When you do, somebody is inevitably going to suggest that you dive into the highest level of testing maturity and skip all of the previous steps. Don’t fall for it. It will be a frustrating waste of time, because you would be building on top of a non-existent foundation. You would end up with a nice roof over some dirt. It is not what you want.
I would like to discuss the natural progression of a mature testing process. Any time that I’ve seen a team take a different path, they always seem to fall-back into this natural progression.
Here are the maturity levels that you are (most-likely) already familiar-with but might not have names for them:
Level 0 : No testing
God help us all. Your customers are your testers and they find your bugs at inopportune times (usually when their work is disrupted by an error or program crash). Your customers start to hate you and you hate yourself sometimes. You will always have a gnawing sense inside of your gut that tells you that there has got to be a better way.
Level 1 : Monkey testing
When I say monkey testing, think of the theory that, an infinite group of monkeys randomly pressing keys on typewriters/computers, would eventually produce all of the literary masterpieces that we have today. It is a funny concept and therefore, a little hard to accept. Software testing usually starts at the same point: get any people to do any testing. Certainly, any testing is better than no testing. “We randomly tested this app for four hours without an error. Ship it!” This approach is a little silly, but you will still get some genuine ROI from this form of testing.
– Anybody does any testing
– If a bug is found, then it is reported to the developer(s)
– Eventually, you will feel a need to track these bug reports in something more sophisticated than your email inbox
– Testing is unreliable and you are not sure how much is really being tested. Some say “lots” and other seem to think it is less than is being reported. Either way, it seems like it is not enough.
Level 2 : Planned testing
Okay, now we are getting somewhere. This seems more scientific, because you are using a plan, therefore you can show what was tested. If a bug slips through, don’t blame me, blame the test. The test gets fixed and the leak is plugged. You quickly realize that you could/should have a dedicated person who executes these tests and is responsible for them.
Level 3 : Organized testing
You feel like you should be wearing a necktie and using fancy lawyer words like “shall”. You have lots of spiffy test plans. You have them organized into test suites and a library of tests. When testing time comes up, you already know the plan, because you are organized. Bam! Take that, bugs! Your testing efforts have saved some back-sides several times now. You can barely imagine how you ever lived without it.
Level 4 : Automated testing
You have tried everything else and you are expanding until you have tried everything. The monotony of much of your testing is starting to weigh on your testers. Many of your test plans are practically carved in stone at this point. You automate those tests so they can be ran very quickly. You make a few mistakes by over-automating some of the tests, but you eventually learn from those mistakes.
Level 5 : Harmony / Nirvana
Your testers focus on manually testing new stuff and developing new plans. Once a new plan is mature/stable enough, someone turns it into an automated test. Your experience tells you when to automate each test and when that would be pointless. Your testers are involved early in the process (like during requirements gathering). The developers are even integrating test hooks into the code so that testing can be done more reliably and definitively. You’ve even tried TDD (test driven development) and have a good understanding of where it provides value and where it does not.
At each of these levels, there will be some significant “lessons learned”. You will try something new, it will work, and you will go overboard with it until you realize that things have gotten out-of-hand. Then, you will reel it back-in. You now know how much is “too much” and how much is “just right”. You are ready to advance to the next level.
Now, try to imagine going from step 0 to step 5. This was what I tried to describe at the beginning of this article. I’ve seen so many teams try it (you probably have too). They just dive-in and start automating away, without a reliable plan or any kind of experience to draw-upon. It is like going fishing with a football bat (There is no such thing as a football bat, but try to imagine how silly it would be). Somehow those darn fish manage to avoid your football bat, like they were born to do so. Well, so do the bugs.
So if you are growing your testing process or just getting started, take my advice: do the steps, in order. Plan to mature at a natural pace. If you skip the middle steps, the advanced steps (that you jumped forward to) just won’t work well or just won’t stick. You will quickly learn that each of these steps is unavoidable (for some reason), just as I have.