QUALITY MANAGEMENT

This post is the final post in my series about what I see as the differences between testing (Part 1 – What is Testing and Part 2 – Testing Activities) vs quality management (Part 3 – What is Quality and 4 – Quality Managment).

Who Owns Quality

The whole team needs to take responsibility for quality. This is a phrase I use frequently to mean that testers do not own quality. In reality, it should not be only the team that owns quality. People at every level have a different part to play. The picture below is the last slide from a talk that Lisa Crispin and I did for the Agile Testing Days webinar series and shows some places where people contribute to quality.

It starts at the organization level, by having psychological safety in place, where the ‘blame game’ is not played, and teams feel that they can experiment without getting punished if they fail.

Many organizations have a ‘Center of Excellence’ of some kind that mandates processes that teams should follow. In my experience, this doesn’t work well since teams that don’t see the value in a process will find a work-around or blindly follow something they don’t understand. I’ve had much more success working with teams to start small, evaluate and adapt getting better with each adaptation. Even when your experiments aren’t as successful as you had hoped or fail, because they are small, you learn without huge cost.

The organization can define the level of quality they need or want but let the product teams define how to get there. Perhaps starting with a basic definition of Release Done. Of course, there are exceptions. Not every team can do exactly what they want if they need to work in conjunction with other teams. They need to coordinate and find common processes. Perhaps they have a common definition of Feature Done, and each team has their own definition of Story Done.

Product management folks need to coordinate their efforts so that the most important pieces of work get done in a timely manner. If teams are fighting over resources and people are working on multiple teams, effectiveness is lowered, and efforts are likely not focused on the quality of the product. Features may seem to be the most important thing, but sometimes teams need to spend time addressing technical debt. Over time, technical debt will erode the quality of the product as it gets harder and harder to fix and add new features.

Business stakeholders, whether they are product owners, or business analysts or marketing, need to understand what it is teams should be building, and how to communicate the ‘what’ effectively. The development teams can then concentrate on the ‘how’ and own those processes so that quality can be built in from the beginning. Demos are one way we get fast feedback from stakeholders to know if we are building the right thing.

Other folks that may want input into the quality are governance – they care about regulatory conformance, or perhaps security. Look around your organization to understand how people in different jobs, with different goals may perceive quality. Start that conversation if you haven’t already.

I believe every individual in an organization should be able to articulate how they contribute to the quality of their products and it starts with you.

Measuring quality

We have many ways to measure the quality of our processes, but it’s much more difficult to measure the quality of our products. I reached out to folks on twitter to see how their teams measured quality and received a variety of answers. Most answers were about metrics based on processes. Only one answer was based on the users, and none considered the value or customer loyalty. The responders seemed to focus on the ‘how’.

A quote from Lisa Crispin’s and my second book More Agile Testing, describes how I think about process quality.

“When you work in a company whose leaders understand that focusing on quality is one of the best ways to deliver business value to customers frequently, your team – and company – are likely to succeed. Frequency and consistency of delivery improve over time. Conversely, if a company’s focus is on speed first, often quality is sacrificed. Technical debt in the form of fragile, hard-to-change code and slower feedback cycles decreases the team’s ability to deliver consistently.”

Examples of metrics that I received from the twitter respondants that fall into manufacturing-based (process) quality are: number of bugs in production (prod), severity of bugs in prod, days since last push to prod, number of new support tickets within five, ten, and 30 days of last prod push, build radiator remains green, no flakey automated tests, static code analysis of codebase is healthy, rework rates are low, number of bugs that get reopened. None of these tell us if the product is valuable to the users, but they satisfy the human need for numbers. They are also easy (fairly easy) quantitative measures to capture but can lead us to incorrect conclusions about product quality. Daniel Kahneman talks about biases and how we misinterpret metrics in his book Thinking, Fast and Slow.

It is important to recognize the difference between process quality and product quality. For example, people often want to measure bug count to talk about product quality because it is simple to do. However, I don’t think that measures what we think it does. I believe bug counts measure how bad our product is, rather than how good – especially those reported by the customer. Let’s say the customer reports 5 minor bugs. Does that mean the quality is good? What if they reported 5 high priority bugs?

There really is no easy way to measure product quality but we can try. For example, Isabel Evans showed this graph a conference, showing two different products that had basically the same functional features. However, they were aimed at completely different users. One group did not care about speed and was afraid of making mistakes (not tech-savvy). The other group valued being able to complete tasks quickly and wanted flexibility in their product.

In this case, value was not a price point difference, but an attribute difference. These organizations were in the same product line but not really competitors since they had a completely different customer base. They based their quality measures on value and user-based metrics for individual customer personas. Both provide value to the customer they were targeting.

Organizations tend to use more qualitative measures for product quality like above based on surveys or feedback from customer support. We can also measure things like customer loyalty – how often do customers come back, or how long to they stay customers. For example, if a company offers a product for free, but gets income from add-on services, customer retention is extremely important.

However, in industries such as health plans and utilities, where consumers have a hard time switching because lack of any kind of quality, measuring loyalty is probably not a good measure of product quality.

In my coffee example, from my previous post in this series, the customers are also different. For example, the person who likes any black coffee for 50 cents, likely won’t buy their coffee at one of the boutique coffee places. The customers who would rather pay $5.00 for a specific taste or a great cappuccino, become loyal customers if they enjoy the coffee. Could we measure how long the line up is every morning for coffee to measure loyalty? Or do the loyalty cards give us some idea? There is no easy way, but perhaps we could extrapolate by looking at line items that sell more to see what customers like.

Do you know where your product stands?  Are you with a utility that no matter what you do, those consumers will likely stick it out no matter what? Or do you have to continually worry that your customers will leave if they have a bad experience? Have you thought about what you can do to influence your product quality? When you look at metrics, don’t go for the easy ones immediately – keep asking the hard questions so you start to track the right information. Ask yourself “What problem are you trying to solve?”

How do we start the quality conversation?

The conversation around quality is a hard one to have. I find that by asking questions about risk is a good starting place. Start with your team, and then expand to other stakeholders you have identified. Ask “What’s the worst thing that can happen?” – that would be the biggest risk.  Ask, “What’s the best thing that could happen?”. That might be what you want to measure.

A good quality strategy should be based on mitigating risks but also should also address reaching for that transcendent quality for your customers (mentioned in my last blog post), if that is an organizational or product goal.

Margaret Dineen did a talk and wrote a blog post about using sliders for quality attributes to start the conversation within a team. Identify quality attributes that are important to your product and then having a discussion using examples to help understand what people mean. Use the sliders to see if people have the same understanding about what is important. It can be very revealing and may open up new conversations. I would go one step further and take this idea to other stakeholders, maybe even your customers to understand what is important to them.

Be inventive – for example, Marco Ravicini created a treasure map for his team. I like that he talks about it as a journey of discovery, and the start of a long discussion. The power is in getting shared understanding about what quality means to your team, your organization.

Final thoughts

There is a phrase I’ve seen on social media which really stuck with me, and sadly I can’t remember who to attribute it to – “quality is woven into the fabric.”  I think this is a great visual, that goes along with our more common phrase of “building quality in.”

Many years ago, I belonged to a quality discussion group made up of people from every industry – manufacturing, oil and gas, health and safety, etc. I was often the only representative from software development. I learned a lot by listening to the conversations, but one of my most memorable discussions was around the cost of quality.

We often think about quality as costing something… extra processes, extra work. But, a lack of quality may be far more expensive, if we don’t consider both tangible and intangible costs. There is loss of opportunity, or erosion of trust and reputation. Once lost, these are hard to get back. Start by analyzing and understanding your risks. That is where your most valuable testing begins.

—

I continually evolve my thoughts around testing and quality, and I hope you do too. I relish every conversation I have on the subject with folks like Paul Seaman or Mike Talks who challenge me and have made me think harder when I write. Feel free to contact me if you want to discuss any of these concepts in this series on testing vs quality management.