Due to COVID 19, all workshops and courses are remotely given for the foreseeable future.

Janet Gregory: Agile Consultant, Trainer, Advisor, Writer, Speaker

Part 3: Testing vs Quality Management – What is Quality

What is Quality

In my most recent blog posts ‘What is Testing’ and ‘Testing Activities’, I shared my thoughts on testing. There is a close relationship between testing and quality, but they are not the same. Testing in some ways, assesses quality, but only to a certain degree.

My intention was to make this the final post in this series and look purely at quality and quality management. As it turns out, I have lots of thoughts on this topic as well, so I divided it into two posts. This first part is how I look at quality.

My preference is to use the term quality management rather than quality assurance. Information from testing can help improve the quality but cannot ‘assure’ the quality is there. When I obtained a quality manager certification from ASQ, I learned that quality assurance was ‘supposed to be’ focused on good processes. However, the term has been subverted and misused, so I don’t think it is appropriate any longer – at least to software development.

There are many different contexts, and each may need a different way of looking at quality. I believe that in all cases, quality needs to be built into the product from the beginning, and I like to make the customers part of the process.


The most popular definition I hear these days is from Jerry Weinberg; “Quality is value to some person.” I like it, but I think may be too simplistic and understates some of the dimensions that we should be thinking about.

In the last few years, Alan Page and Brent Jensen have been talking about their ‘Modern Testing Principles’. and their 5th principle is: “We believe that the customer is the only one capable to judge and evaluate the quality of our product”. Like Jerry’s, it is a simplistic quality model. Personally, I need something more meaningful so I can focus what the quality strategy should be for the product I’m working on.

If we go a bit further back, W. Edwards Deming defined quality as “Good quality means a predictable degree of uniformity and dependability with a quality standard suited to the customer.” Again – based on the customer. Many of Deming’s 14 principles in his book Out of Crisis are still applicable today and bear looking at further. For example, he talked about building quality in and ceasing dependence on inspection.

Dan Ashby wrote a post about Crosby’s 4 absolutes of quality showing how they are still applicable but need to be adjusted to fit into a software context.

A few years ago when I was preparing a talk about quality processes, I had a conversation with Isabel Evans, and she pointed me to this article by David A. Garvin  It got me thinking more about quality really means. I share some of his ideas below but encourage you to read his full paper. You can also read this article which adds in his eight quality dimensions.

People talk about quality as if there is only one kind – all encompassing. Most of what we measure is how well we adhere to our processes – what I call ‘process quality’. To me, ‘product quality’ is about the quality of the finished product – what our customers experience.


Product quality can be looked at from many perspectives. The customer’s perspective is only one – perhaps fitness for use or meets their needs, and different customers have different needs. We also need to consider other stakeholders who have an interest in our product’s quality.

David Garvin talks about five approaches to quality and I use the diagram below as one way to look at them.

Garvin Quality

We have different lenses in how we view quality. We can look at it differently depending on our circumstances. I start with the inside circle since that is the easiest to think about, and then work towards the outside.

Manufacturing-based quality

If we equate manufacturing to our development cycle, the emphasis is on the practices we follow and conformance to requirements. Excellence is equated with meeting specifications. The focus is on defect prevention and limiting rework. This is often where many teams spend their time building it right and measure the quality of the process. Many teams target testing to this perspective.

We need to recognize that software development is not like manufacturing which builds the same thing over and over. We are continually building on to what we have, changing and adapting.

Some processes that support development (manufacturing) quality are: TDD (test-driven development), coding for maintainability, code reviews, continuous integration, exploratory testing or even conforming to the definition of Done. We are measuring process quality and we answer the question – “Are we building it right? “

Product-based quality

Garvin talks about product-based quality is about the quality of the attributes, the pieces that put together to make the whole. The thought is that better ingredients make better products.

Because this approach to quality reflects the quality of attributes that a product contains, and attributes are considered to be costly to produce, higher-quality goods will be more expensive. If quality is viewed as an inherent characteristic of goods, higher quality means better performance, enhanced features – things that may increase the cost of the product.

Paul Seaman and I had a conversation about this and used this metaphor to think about it – an ordinary cook with good ingredients may not have a good outcome, but a great chef with ordinary ingredients may produce something magical. We need to understand what our product is and how it is put together.

Some processes that teams can do to help support product quality are: ATDD (acceptance test-driven development) or BDD (behavioural-driven development), as well as testing quality attributes such as security, performance or reliability. We answer the question – “Does it work as expected (or desired)?”

User-based quality

The user-based perspective is what we see used most often to talk about quality, and it is highly subjective and highly personal.

It assumes that consumers possess sufficient information to evaluate product quality. If they do not, they will rely on other cues when making that assessment. Let’s consider a cup of coffee. I prefer a simple black medium roast coffee to a well-made cappuccino, but my sister will take the cappuccino every time. What does this mean to you? Who is your consumer? Who is evaluating your product’s quality? Do you need to satisfy the majority of consumers? Or target a specific group and satisfy that group only?

coffee - carol plain coffee

Some of the processes that I think help support user-based quality are: user experience (UX) designers working with customers to learn what they want, ATDD, testing using customer personas, A/B testing, accessibility testing, observability and running the analytics on production usage. We try to answer the question “Is this what our consumers want?”

Value- based quality

Value is defined in terms of cost and price. Does the product provide performance at an acceptable price?  That cup of coffee – some people only want to pay 50 cents while others will pay $5.00. That is 1000% difference in price. Is there that much difference in the actual bean? Or the roasting? … Or is it something different?

Marketing specialists are often the folks who care about this. They may do research on price points, give customer surveys to understand what customers think, run analytics on number of products sold, or determine what the profit of specific features might be. These are tests that validate hypothesis and help organizations understand how people use their products and what they find value in.

Product management folks also care about value. They must find that price point that gives value to the customer, but also makes them the profit they need to stay in business.  The question we need answered is “Does the customer find good value in our product?”


I’ve left transcendent quality until last. Garvin describes it as ‘innate excellence’ – universally recognizable, a mark of uncompromising standards and high achievement. It’s hard to quantify or define, but you know it when you see it. Your senses tell you. Perhaps its the ambiance of that great cup of cappuccino you keep going back for.

When organizations enable teams to create something special, to go beyond the norm, that’s when you get transcendent quality. An example might be some unexpected amazing graphics for a new game. I once worked on a system that was replacing one that was no longer supported. When I asked about how the users liked it, I was told “they loved it”, because it followed their work process and not the other way around. A little bit of transcendent quality.


If I got you thinking a little bit more about your definition of quality, that is a good thing. It is not a simple conversation so many folks don’t talk about it. In one workshop I facilitated, I asked participants to come up with their definition for quality of a specific product. After you’ve read this, go back to your team and ask that question. Check to see if the definitions cover how well you develop (process quality), or are you really defining product quality.

In my next post, I get into quality management – who owns it, measuring, and how to start the quality conversation to put together a quality strategy.

You can check out some of my thoughts from an old blog post about quality and its cost I wrote in 2010. P.S.  I still drive a Mini Cooper although a new version with more bling than I wrote about in that post. If you do read it, I suggest you read the comments too.


8 comments on “Part 3: Testing vs Quality Management – What is Quality

  1. Albert Einstein – “If you can’t explain it simply, you don’t understand it well enough.”

    Thank you Janet for explaining and illustrating “what is quality”.

  2. Interesting post, Janet. And I agree very much that quality is something we should not avoid talking about!

    So here’s a philosophical reply – a critique if you wish 🙂

    I think the product-, user-, and value-approaches are one and the same. Or at least they are variations over the same theme.

    For example in your case with the chef, the quality “magical” must be user based and subjective as this “magical taste” must logically be experienced by someone before we can declare it. And then there’s always the risk that someone will disagree.

    Value (even monetary) is also subjective. Economy is the science of generalizing it.

    So what can we do with quality? We can approach it analytically or from a manufacturing perspective only to a certain degree – quality has to be experienced in the real: Tested, or experienced in production. Without experience, quality is only theory.

    I believe this is what Weinberg mean by his statement: That quality is subjective and experienced, and any generalization is complex.

    This leaves us in a difficult situation: No man ever steps into the same river twice, as Heraclitus said. Similarly, quality is never the same.

    Immanuel Kant joined us philosophically by identifying a set of universal categories of terms. I actually see them as the missing link to talking about quality. To Kant, “things” are quantifiable, qualifiable (e.g. colour), relational, or modal (concerns our existence). (Kant does not discriminate between concepts and physical objects, hence “things” in quotes).

    He said we should transcend ourselves from the immediate observations and intuitions we make as they are always only imagined representations of reality. The reality as it is in inaccessible to us. (That’s why he didn’t discriminate between the two.)

    Recent neuropsychological research has confirmed this idea of his.

    And so even the transcendent approach is the same as the other three: Quality must be recognizable to make sense at all! The magical taste as well as the user experience and perceived value must be recognized.

    The solution? Let’s keep talking about it – and making and challenging definitions of quality.

    By the way, I don’t like the manufacturing idea of quality. I actually think it’s wrong as it represents the false idea that intention == outcome. It know it has driven industrialism and that you can often get a long way writing beautiful and readable code, but it’s time to move on. After all, code is not manufactured, but producs of creativity. Quality happens.


    • I don’t particularly like using manufacturing as a base either, but it gives us some ‘words’ around quality to start the conversation – as you said, challenging definitions. It’s very difficult to measure quality when we say it’s valuable to some user…

      At the same time, I would hate to think that we measure how beautiful the code it, or how creative we were when we wrote it. It is really is the outcome that matters, not the code itself. I could write the most beautiful code, but it is useless to anyone that matters.

      Quality is difficult to define. No question. I hope to get people to start having the conversations and really think about what it means.

      PS. I’ll have to read some of Immanuel Kant’s writings. Thank you for that.


  3. These different definitions of quality are so important! Many companies neglect quality in their internal software development because “we are not customer-facing”. Well, not only there is always a customer (it does not need to be someone paying for the software), but there are many other ways to build quality. They all have the potential to reduce costs, maximise revenue and provide new opportunities for the company either directly or indirect.

    Thanks for showing this in such a straight-forward way, Janet! You’re always an inspiration.

    • I’m glad you found it useful. Sometimes people forget that they have internal customers. There is “always” a customer, and usually more than one. Sometimes we have to look for them, but I think it is important to do that. – Janet

Leave a Reply

Your email address will not be published. Required fields are marked *