Measuring Testing Success

Over the years, we have seen some very successful testing performed on large IT projects, and we’ve seen testing  that wasn’t very successful.  Testing is absolutely necessary, and successful testing is absolutely critical.  What are some of the more challenging problems we’ve seen with testing?

• Testing is executed, but unsure if problems are fixed
• Hard to measure whether testing round was successful
• Begin integrated testing with an unprepared system

In a typical integration testing round, which I think we’ll all agree is the round that we most look at to see if we have a fully functional system, the teams usually move through scripts with a pass or fail for each line item.  And then we’re left to wonder if the problems are really getting fixed between rounds.  We’re aware that a round of testing is done, but how did we really do?  And we’ve got to wonder if we really came into integrated testing having done as much due diligence during unit testing as possible.  

We’ve developed a fairly simple way to help you make sure you get the most from your testing and the most from your team as they prepare and conduct testing.  This process helps address all three of the challenges we listed above:

1. Count the number of line items for EACH integrated test script.
2. Tally the number of passes and fails.
3. Do the math – and assign a grade.
4. Make the team know they are to be graded.
5. Tally the pass/fails for each specific team across all scripts.
6. Let the team fix and retest – then re-grade!
7. Expect an A+ from everyone and every script.

While this is a very simple approach, you can see that it sets expectations before integration testing begins – and this will encourage your team to fix as much as they can so they pass as many items as possible.  It gives you numbers to determine your level of success.  And it insures that each issue gets resolved with you ending up with scripts that can be executed flawlessly.

We know that no amount of testing is ever “enough” but we hope that this tip was helpful and look forward to hearing your personal experiences with testing.