Search for:

Speedometer Confusion

I follow Seth Godin as he has incredible insight and ability to simplify things down to a few key words.  His post, Speedometer confusion is one of may such posts.

Too often, being busy, is confused with productive.  We set metrics like how many defects found, how many test cases run and how many hours tested. But what exactly do these metrics mean to software quality? Looking down at our speedometer, we may think we travelling at 200km/hour for the last 8 hours covering some 1600km.  But what if when we check the GPS, it shows we only covered 100km.

We are often wary of establishing effective KPI, worried about what these would really reveal, or worried management may not understand. Opting to track “safe” KPI that show some form of effort rather than effect. Rather than number of test cases, why not measure percentage coverage? Rather than defects found, why not measure defects missed? Rather than hours spent testing, hours saved?

It is nearly impossible to improve, if one fails to take an accurate measure of the current situation and set goals for the desired outcome.


Following on from SDLC strategies, the second group or area to focus on when attempting to reduce the total amount of testing is ways to reduce the number or Test Iterations or levels.

Testing Levels

Every organization has their own st3pps in their SDLC and organization structure. This often depends on the defined roles and functions in the organization and their development approach. A large waterfall based organization may have dedicated security, identity and load testing teams, while a smaller agile organization may have these functions fall under a single team or person. In every organization, there is a struggle between finding the most efficient process vs budget, personal agendas and overcoming inertia. “Its always being that way”, and “its done by a different team” does not imply optimized.

Unit Testing

With JSON services not having WSDL or Schema, many organizations have had to find new ways for Development and Testing to work together to ensure the correct coverage. Developers and testers may work off the same service description document, but how accurate is the document and how well does it match the developed services? What if a developer added or missed some functionality when coding? How does Testing even know what services are there for testing? Group development like Agile or Extreme programming may help, yet requires testers to have a fair understanding of JSON.

A second challenge is how do you gate or ensure that developers have done at least some level of due diligence before passing the code drop over to Testing to begin testing. If the defect density is to high, and code needs to be heavily reworked, the entire test cycle is wasted. If previously found issues are not suitably addressed, the same retesting is again a waste. Release management can take significant amount of time and resources.  To prevent this, various organizations have tried different approaches. From formal sign-off’s, to change documentation or release management and change management tools. Another approach, is to have one team develop the basic unit test, and this test be used as the gating process. Testing then takes these unit testes and expands them out to include far more intensive coverage. In SOAP its most often the testers that develop the unit test, but with JSON, it is now sometimes the developers that develop the unit test together with the service. Sharing tools and test cases only makes sense, as the developers require some way to test the code they developing in any case. Handing it to testing to expand is good practice, to ensure that “fresh eyes” are used. But what if a developer adds code not in documentation and does not hand a test case over either? Is this risk acceptable?

Whatever the approach, the objective remains the same, from release 0.1 to GA, how can one reduce the number of test cases. The concept also remains the same – Shift as much Testing as possible left, or earlier in the SDLC right to testing while developing and making sure that “code drops” have a certain amount of maturity.

Integration Testing

Integration testing of old was usually a different team to that of Unit testing. As per the post on Continuous Testing in Agile, if using and automation tool, the test cases should be the same. Integration testing, should just include expanding the test cases developed in Unit Testing to include things like encryption, cookies, identity, performance of each service (and enablers) etc. These individual test cases then linked for automation in what we call chaining. Integration testing often then be done concurrently with unit testing, much earlier in the SDLC.

Using the right automation integrated tool, that supports encryption, identity, performance and as many of your integration tests in one, enables the elimination of distinct test cycles for each of the Integration tests. SOAPSonar’s ability to do functional, security, performance and load testing using the same test case, is the number one reason customers say they selected it. That said, Performance of a individual service is something I recommend be part of a unit test and a functional requirement.

Consolidating integration tests into as few as possible and shifting as much as possible into functional unit test cycle can greatly reduce the number of iterations. The impact of removing a single test iteration can clearly be seen in service plan costing model. This is not about increasing risk by skipping systems tests..

Systems Testing

Systems testing requires a certain amount of code to be developed and tested, and the environment to be built.  Moving it too early in the SDLC can actually add to the testing needs. On the other hand, if the integration and unit tests are all automated, and completed successfully, systems testing becomes more about the environment and less about code. Finding code quality issues at this point can be very expensive and troublesome to fix in time.

Again if all the unit tests, and integration tests are automated in the same test case, and if the tool offers the ability to do systems test eg geographically dispersed load testing, the effort required during this iteration can be greatly reduced. A note on performance vs load testing. Too many times I hear that performance testing is left till Systems Testing and suddenly its discovered days before release that there are performance issues. How a service performs should be tested far earlier. At this stage, Load testing should be establishing that the environment (servers, network and other infrastructure) is suitable and not a individual services or enablers performance.

Using Virtualization or Simulated Services to stand in for services either unavailable or that cannot be load tested can be very useful in enabling earlier Systems testing. Simulated services are key trouble shooting tool to eliminate environmental factors like network, cloud, data integrity and servers.


Acceptance testing is were you suddenly discover if all the work was correctly focussed. Have your Developers and Testers spent all this time developing and testing the right requirements?  NIST lists business requirements as the number 1 issue in software development. Often the users and sponsors are not technically minded and extracting the requirements can be very difficult. Bringing Acceptance Testing in earlier to review what is being done can greatly decrease time and effort spent in incorrect requirements delivery.

In web services, acceptance testing is usually heavily weighted on the client side usability and visuals. The need to see an example and not just a jpeg. This is another benefit of Simulated services. By creating a basic simulated service earlier in the SDLC, the GUI team can begin work. This GUI can then be used together with the simulated services for earlier acceptance testing and to compare to the end services built.


Regression testing between releases is an important way to minimize the testing scope. If a regression test of a service shows no code change between release 0.4 and 0.5 then those services were not changed since the last test cycle and need not be tested again. lets say a regression test says 30% of the code was had no changes, using the service plan costing model, how much testing could that save? That said, if they automated, it may be of little benefit not “pushing that button” and retesting them.

Where regression testing really shines is post launch. The maintenance phase of software often costing far more than the development phase. Yet if your automation test cases developed during the development cycle can also be used for regression testing during the maintenance cycle, the effort is greatly reduced. This “continual testing” post production and even API monitoring is rapidly growing area of focus for many organizations as more complexed meshed applications are being developed.

A note here, so often I hear that security or some other team wont allow for automation. The impact however of a single systems test iteration not being supported can cause any automation to be useless on production environment and prevent any regression testing using the previously built test cases.


There is a lot about automation in this post. Every day I hear of benefits and issues with automation. Many organizations try apply automation to their old process and find little benefit and become discouraged. Alternately, a tester involved in load, security or some other single test iteration, seek a tool to do just that iteration – sometimes just for control of their environment, breaking the larger benfit by doing so. The key to successful automation is and end to end approach from the start of the SDLC through. You may choose not to automate the whole process, or use other tools, but the design and application of automation needs to be done with the entire cycle in mind.

The next post will be on ways to reduce the number of unit tests.


Calculating Percentage Coverage

I wanted to discuss some confusion of percentage Test Coverage. I have noticed that different organizations calculate test coverage very differently.  This can be very confusing when using contractors, off-shoring and simply comparing best practices. Lets say you have a simple service that returns Name, Phone Number and Address and you asked to create test cases for 100% Test Coverage. What exactly does that mean?

Would a simple unit test entering the following be considered 100% Coverage

  • Bob Smith
  • 555.555.5555
  • 55 street
  • City
  • QC
  • M5M 5L5

Or would you need to break the service down into each of its functions. Name, Phone Number Street, City, Province, Postal Code. Testing each of these independently?

But how many test cases do you need to perform, for you to consider it 100% coverage. Lets take Postal Codes. Would a single Postal code be considered 100% coverage? Or would you need one from each of the 18 starting letters ( Y, X, V, T, S, R etc)? Perhaps you require some random number of say 10 or 100 postal codes? Or do you need to enter every defined Canadian Postal code.  Lets consider testing name function, how long a name does the app need to support, how many names can a person have, what if we include title, what if the persons name has a suffix.

What about negative scenarios? Do you need to test postal code that does not exist, or one in the wrong format before one can consider the test coverage to be 100%? With space after first 3 letters, without space, or with a hyphen. What about letter number letter or what if all letters, number or some other possible combination? How many of these negative scenarios does one need to run to say you covered 100%?

What about testing these functions as they relate to each-other or as this service relates to other services?  Do you need to test that a Postal Code starting with letter V, is not used for a city that resides in Quebec? Do you need to confirm that this address service when used in one chained request, responds the same was as when used in another? So often I hear of companies unit testing services as they developed, but never running a final systems and integration end to end test. What if one service requires that postal code to have a hyphen and the other a space?

Understand if your organization is manually testing a service, entering even 18 postal codes will take significant time directly impacting costs. Entering all positive, negative scenarios including chained services is just not feasible. Does increasing the number of test cases actually effect the percentage coverage, or is a single test case enough? All the possible boundaries for a simple service like postal codes could result in a large number of tests. Does testing  each service once, without considering all the boundaries and negative scenario’s constitute 100% coverage? More importantly perhaps, is when QA testers give a percentage coverage, does it really mean the same thing to the everyone?

I would like to invite everyone to weigh in and share their thoughts on the subject. Please select and option and comment if you will below. So far the majority selected test every function once. So I broke this into boundaries and positive and negatives to see if we can get further clarification.

***Please note The form is submitted privately and is not automatically published. If you wish your response published, use the comment link at the end of any post***

Warning: strpos() expects parameter 1 to be string, array given in /home/content/13/11164213/html/ST3PP/wp-includes/shortcodes.php on line 193
[contact-form to=’[email protected]’ subject=’percentage coverage’][contact-field label=’What does your Organization consider 100%25 Test Coverage?’ type=’radio’ required=’1′ options=’Whatever We have Time for,One Test for Each Service,Test Each Function of the Service only once,Boundaries for Each Function,Both Positive and Negative Boundaries for Each Function,All/Many (Data Source) in Chained Workflow’/][contact-field label=’Comment’ type=’textarea’/][contact-field label=’Screen Name’ type=’name’/][/contact-form]


For those of you who attended my TASSQ presentation, this post is not new, but for those that did not, I wanted to so a post to give you some background.

The example we used at TASSQ was;

“If you have an assignment to test 500 services, each service has 10 functions and each function requires 4 tests, upper boundary, lower boundary, and 2 negatives. Using service plan costing, the total number of test cases you would need to test is then 500 x 10 x 4 = 20,000.”

Now before you say, there hundreds of ways to reduce that number, let me agree and say, yes there are many ways we could. But lets leave that discussion for other posts, or another presentation (was actually something I was going to speak to) and pretend that this number is is the result AFTER all optimization is done. The Reducing Amount of Test Coverage series of posts will address ways this can be done.

Service Plan Costsing

The TASSQ numbers were numbers in the second column, I got polling the audience, and I believe are pretty conservative.

  1. Number of Release Iterations – This is how many times you get a code drop from Development that is tested by QA. Each time, QA runs through a series of tests, be the manual or automated.
  2. Integration Testing is testing integration with Identity systems, 3rd parties and existing systems.
  3. System Testing includes testing the system as deployed. Testing the environment. Load vs performance, DR, preproduction, and production etc.
  4. Time per test case, is a breakdown of time taken to run a single test
  5. Hourly Rate all in, needs no introduction – we tend to have a slavish focus on this number which often translate into salary cuts.
  6. Tools and Infrastructure costs is the capital expense for buying software etc. to test
  7. Training – far to often Zero
  8. Additional overhead, any number not previously in there, like recruiting costs etc.

The result is a project cost of $1,473,500 which would take 10 resources at 8 hours a day 450 days to test.

Now for the not so obvious Truth about Testing.

  1. In Column 3, we changed 1 number, test time, to understand the effects it has. If you could save 1 minute a test case, the multiplier effect translates this into $160,000 savings or some 11%. How do you like that for a reason for optimization or Automation. Time and Time again I am told, “but we can do the same thing in-house without buying tools” and time and time again, I ask, “but does it take more or less time”. I heard QA staff complaining about no training, and yet, a course would be cheap in comparison. More Importantly for some we cut 50 days off testing time!
  2. In Column 4 we dropped one test cycle for a saving of $120,000 or 8% or 37 days. Remember the last time development gave a code drop that was not fully baked? Why do people use different tools and test case’s for functional and performance tests, when they could wrap them into one?
  3. In Column 5 we reduced the hourly wage by $5/hour. The effect, $180,000 or 12%, yet time remains constant…. Not as much effect as you thought perhaps?
  4. Column 6 we tripled tool costs, in an effort to reduce testing time or release cycles. The effect is some 4%


So regardless of if your business uses service plan costing or not, it is a valuable tool for understanding and dissecting QA costs. For focussing attention on the matters that really effect costs. In fact, I believe KPI and pareto analysis of these KPI is important for any QA manager. I always like to hear ideas that reduce testing time or test cycles, so please, feel free to share your thoughts.

TASSQ Presentation

I want to thank everyone for their participation and input in last nights TASSQ presentation. Although its was a very simplified scenario, using a LOT of Averages, and the fixing of the 20,000 test cases required limited options, the limitations were set to demonstrate the impact of change more clearly. If we had more time, we could have looked into changing coverage ratio’s and changing the number of test cases required per Release or test iteration. I encourage everyone to do just this, using the own projects, to see what personal insight it will provide.

I have put the slides and spread sheet used for My TASSQ presentation on Google Docs for view or download.

Much of what was presented is in more detail on this blog, which you can subscribe to receive a monthly newsletter of updates. I also encourage all to make comments and share thoughts. Only a screen name is displayed.

Thank you again, and I hope you found it as interesting as I did.

ST3PP Presenting at TASSQ in March

Join us at TASSQ this March the 25th in Toronto at the The Albany Club – King & Church Streets 5:30 pm networking 6 pm dinner 7 pm presentation. March’s presentation will be done by ST3PP’s me, Ivan Flowerday. Visitors and new members are always welcome at TASSQ and attendees include anyone who has an interest in software quality, or software testing, whether they work for a large corporation, a small software house, or whether they are independent consultants.

ST3PP’s mission is to Align the domains of Tools, Process and People, to achieve excellence. At TASSQ, I will be sharing some thoughts and ideas collected by meeting with a number of QA and Development organizations. The session will also be interactive, with attendee participation.  Although it will cover all 3 domains, no specific organizations Tools, People or Processes will be used. The intent being to generate ideas or thought that members can further investigate, and perhaps apply in their teams. Similar to much of ST3PP ahead newsletter.

To give you some idea of what I will be discussing you can read a detailed look at service plan costing

We look forward to seeing you there and please stop by an introduce yourself if you have not already.