KPI – Identifying Areas in Process that Requiring Tuning

Both Six Sigma and CMMi focus on creating and managing a feedback loop. Assessing the current status in your organization and then identifying areas for improvement.  To do this, KPI or Measurement Objectives are needed to measure the effectiveness of your organizations Tools, People and Processes. Measuring details like Defect Density, Test Coverage, Total defects, reliability etc. Then collecting the relevant data and analysing these measurements to provide insight into the the results, set goals and track progress.

One of my favourite tools or methods to visual and analyse KPI is via Pareto Graph. Its simplicity makes the 80/20 rule stand out to any level of an organization. In the above example the first 7 groups of defects is clearly well over 80% of the all defects. The impact of addressing the others being minimal, while a small change in the first 3 could have major impact to your organizations quality.

Potential issues with Pareto Graph.

Although extremely helpful in identifying areas that need to be addressed, there are some potential pitfalls on relying on Pereto analysis too much. These include

  1. It is important to define and group the categories correctly. They need to both be precise, so that there can be little doubt as to which category the defect fits into and they need to be of a suitable granularity. Making larger groups, can make addressing the sub issues in the groups difficult, yet making too many sub-groupings can result in loosing the visual effect and intent of the pareto.
  2. Selecting appropriate time periods is important. Measuring only a single release cycle or team, can result in wild fluctuations in results. Consolidating multiple teams, releases etc can cloud visibility into a particular team or release issues.
  3. If changes are made to the Tools, People and Process, to address the issues identified in the  pareto chart, adding them to the same period will not offer any insight into the effectiveness of these changes. These changes need to be compared to a previous baseline. Baselines however can become rapidly dated.
  4. Pareto analysis is draws focus on areas of greatest need, but a laser focus only on those areas, can rapidly destabilize others. Some of these other fixes could be potentially low hanging fruit requiring little effort to resolve.
  5. Although Pareto analysis identifies common areas that may require focus or consideration, it does not supply solutions.

Poll

What I have learnt is that when someone sits down and actually gathers the information and generates a Pareto chart for analysis, they are almost always surprised by what is revealed. For benefit of all,  I wanted to take a short Poll.  First off, looking at teams that do use pareto analysis and teams that don’t.  What I would be interested in seeing, is if these teams see common or different set of issues. In other words, does doing a pareto analysis change peoples list of most common defects. The results will be collected privately and consolidated by me, and posted at a later date. Comments at the bottom of the page posted before hand for public, but comments in the poll kept private unless you stipulate in them you wish to be mentioned.

Please take a minute to weigh in and help others ST3PP ahead.

[contact-form subject=’pareto analysis’][contact-field label=’Screen Name’ type=’name’ required=’1’/][contact-field label=’Does Your Organization Use Pareto Analysis’ type=’radio’ required=’1′ options=’Yes – we review them frequently,No – I have never seen one for my organization’/][contact-field label=’1 – Most Common Defect Group’ type=’select’ required=’1′ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’2 – Second Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’3 – Third Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’4 – Fourth Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’5 – Fifth Most Common Group’ type=’select’ options=’Business Changes – Missing or added business functionality,Design Defects – Missing or incorrect architecture requirements,Developer Defects – Missing or defected functionality in code,Environmental Defects – Versioning%26#x002c; enablers or other other site issues,QA Defects – Incorrectly identified as a defect,Other – Please describe in comments’/][contact-field label=’Private Comment To Me’ type=’textarea’/][/contact-form]