Steering Product Quality in Agile TeamsHet sturen van product kwaliteit in Agile TeamsKontrollera produktkvalitet i agila teamProduktqualität in Agile Teams steuern

Few months ago I published a blog on Measuring and controlling product quality for iterative large(r) projects. I made a promise to blog about how we steered product quality in Agile teams, using Fault Slip Through. The approach is to reduce quality risks by deciding in the planning game when and how to invest in Quality. Let’s take a look how you can steer product quality in Scrum teams.

What is Fault Slip Through?

Fault Slip Through measures the number of defects found in later test phases that should have been detected earlier, when finding that defect earlier would have been more cost effective. Using Fault Slip Through this way helps to quantify product and process quality performance and to reduce costs. And, since defects found late in a project often lead to disturbances and can have impact on delivery and release dates, the measurement also can be used to shorten delivery time.

Analyzing the Fault Slip Through metrics tells you a lot about the current and expected quality of your product. We’ve used Fault Slip Through in large iterative software development project, to improve the quality of products before release, thus reducing maintenance costs. With the Fault Slip Through measurement, product managers could decide how much time and money to invest in reviews, testing and defect prevention.

A paper presented at the Practical Software Quality and Testing conference and the blog on how testing drives quality give some more background on Fault Slip Through and how to use it. The overall view on measuring quality is given in What drives Quality, which is based upon the work that I have done at the Software Engineering Institute (SEI) on Building Process Improvement Business Cases Using Bayesian Belief Networks and Monte Carlo Simulation.

But we’re agile. Can we also steer product quality?

Of course you can! I have also applied the Fault Slip Through metric also in Agile and Scrum projects. Our approach was to discuss quality risks with the team in the planning game and identify User Stories that would potentially have many defects. The reasons for the many defects could for instance be that there was much uncleanness on the required functionality, or that code needed to be adapted and refactored extensively, or that it would be difficult to test and validate the functionality, or that team members had very little experience with the required functionality. Where the quality risks of defects slipping through became too high and needed to be mitigated, the team and the product owner discussed what could be done to reduce the risk, and defined engineering tasks for the needed activities. Below some examples where the team and the product owner discussed quality risks and fault slippage, and look at how that helped to steer quality with agile.

In a planning game, a user story  was clarified by the Product Owner. The team members defined all the technical tasks, based upon how they normally develop their software. Design, coding and testing activities were written down on cards, and added to the sprint backlog. In my role as quality manager, I started the discussion with the Product Owner and the team about the expected quality.  The user story clearly described the needed functionality, and the customer wanted this to work properly, so the he made clear that he expected that there would be no defects in the delivered software.Since the team had planned time for testing, this looked feasible. Then I asked the team if they expected any quality risks for this user story? The team members made clear that this was similar functionality to what had been developed before by them, they were familiar with the product, so they expected that they would be able to design and code the functionality without any defects. At that point, the team looked at the amount of hours planned for testing. Given the low risk of defects for this functionality, they decided that module testing, using Test Driven Design, and some freehand testing should be enough. The number of test hours on functional testing and system testing could be significantly reduced! Basically, the team and the Product Owner applied risk based testing already in the planning game, to decide about the amount of testing that was needed, based upon the risk that defects .

In another planning game meeting, the Product Owner described a user story which contained new user interface functionality. The team discussed the testing approach, and started defining scenarios that users could do with the new functionality. Testing hours started growing, until the Product Owner stopped the team and made clear that he had made arrangements with a friendly customer to look at the new functionality, and provide feedback. He expected that based upon the customer feedback, the functionality would need to be changed anyway. In stead of defining many testcases, he proposed to invite the customer to the sprint demo, and arrange time in the next sprint for the customer and team to sit together and adapt the user interface towards his needs. The team proposed to develop a flexible design and coding solution, that would make this way of working with the customer possible. That would need more hours initially in this sprint, but save many hours in next sprints and in maintenance. Given that the product owner was responsible for both development and maintenance, he saw the business benefits of this solution and approved it.

1. Deciding to invest in Quality

It’s often difficult to convince a product owner that it will cost him more money to get the software quality that is needed. What helps is to quantify the quality costs, by showing how many defects are expected to be reported by customers after release. It turned out, after doing quality risk and fault slippage estimations in agile planning games for several months with multiple teams, that only 10% – 20% of the user stories have quality risks and would require additional quality activities like pair programming, reviews and inspections, and testing to prevent faults slipping through. Given the high costs of defects in maintenance, it’s easy to make a business case to find those defects earlier. There is sufficient industry data available, for instance on the business benefit of reviews. Another great resource on costs and benefits of quality is the book The Economics of Software Quality by Capers Jones.

Conclusions

It was much easier for the product owner to decide whether or not to invest time and money in quality, being aware of the potential product quality risks by fault that would slip through to later testing or to the customer. Addressing quality in the planning game, before any software was written or tested, turned out to be a very effective way to prevent defects and improve quality.