Saturday, January 12, 2019

Eff the user

This idea has been running in my head for a while, and every so often I'll stumble upon a reminder to write something about it, and then have other things to occupy me. The most recent trigger was a talk by Joel Montvelisky in a local SIGiST conference where he spoke about the changing role of the testers. The subject is my objection to one of the modern testing principles.  I'll admit - I really like them. That is, with the exception of principle #5: "We believe that the customer is the only one capable to judge and evaluate the quality of our product".
I don't like this one a tiny single bit. I think it's a false statement and it drives wrong team behaviour and goal displacement. Worse, this idea seems to be ubiquitous in the industry, or at least wildly spread, as some of the variants of similar ideas out there show. I think that #5 is one of the most carefully worded expressions of this idea (as one cannot dismiss it with "a tester can't be a customer proxy\champion\advocate") So I'll stick with it when thinking through my disagreement. 
On the face of it, the principle is very solid - if we accept Jerry Weinberg's notion of quality being value to some person (and I like Bach's addendum to it), it only makes sense that the person who is getting the value is the best judge of it. 
Except when they aren't.
I am not claiming here that a product that is not used or bought by any of its target audience can claim to be "of high quality", but simply that there are cases where evaluating the quality of a product is something that the customer is not capable of doing. 
Imagine that you are getting an OS update to your phone. It improves memory management, fixes a security flaw and provides better logging in cases of a crash. Are you in any position to evaluate any of this? Would an average phone user be in such position? All of it brings value - better logging means faster debugging cycles and faster hotfix releases, a security update reduces the likelihood of the phone being hacked and the memory management improvement will mean you could run more applications in parallel and avoid lagging. But how do you even notice any of those? You don't see the crash reports (and there are other ways to improve response time), the security update is like an insurance - you only notice it if something bad happens, and  your phone has enough memory that you didn't notice any problem with the previous memory management scheme. In all of those examples, there are simply too many layers of indirection between the actual property and the perceivable result and the quality of the product, if assessed by the customer, must be assessed by a proxy measurement. In some cases, the customer might notice if you did a shitty job, but the distinction between "barely good enough" and "excellent" is not visible. 
Then there's the ambiguity about "who is the customer". Consider a bug fix that eliminates a scenario where the user is charged for only 3/4 of the actual use. By fixing it, we are taking "value" from the product's user, but adding value to the vendor. While such condition is extreme, there are more examples when one looks on B2B products - The full disk encryption scheme used by our IT department makes our computers freeze for a second every now and then, but provides the decision makers (who don't use their PC as intensely as other employees do) the peace of mind knowing that data could not be leaked by simply stealing a PC. Customer value? high. User value - not so much.

Now, to goal displacement and wrong team behaviour. A simplistic understanding of this principle drives a lot of focused into perceived quality and requires advanced logic-fu to justify any kind of improvement that isn't a new feature. Take for example an architecture improvement for enabling our product to scale to need. We can go with the current architecture for a year or so, and probably throw more hardware on it for the next five years, so, no real benefit to the user, let's not do that. The rule-fu required would be something along these lines:  "If we don't do it then we can probably patch the system for 5 years but it will gradually slow us down as we deal with issues as they appear, and will also be exposing customers to issues caused by this load. In addition, in five years time we'll have more components relying on the old architecture we want to replace, so the change will take longer and be more risky and complex". This sort of logic is not trivial and far from immediate, if we train our instincts to zero in on the customer satisfaction we'll have a tough time coming up with such scenarios, let alone justify such future-looking actions in face of a well trained gut-feeling. Most teams don't need another thing applying pressure and bias towards perceivable features. True, if we insist, everything we do has an ultimate impact on the customer, but sometimes the impact of good engineering practices is several times removed from the actual impact, and it will usually be only one of the factors affecting that specific property of customer value. Imagine two similar SaaS products, both have the same capabilities, a similar size of customers and are distinguishable only by minor features that might appeal more to someone's personal taste. One product is built by a team of 10 highly skilled engineers that are running a tight ship. The 2nd is maintained by 100 people who work frantically to keep things afloat. The cost of operation for the two products is similar - The top notch engineers are getting payed 10 times as the less skilled ones, and the robust hardware and hosting services used by the first team are as expensive as the cost of throwing cheaper hardware  to gap for design problems in the second team. The perceived quality is the same, both companies can scale at roughly the same pace with roughly the same resources - is the quality of the two products equal? They are supplying the same customer value in different means, only the 2nd team is making it really difficult for their operations team.
I got a response for similar claim from Joel who said "The operations team are your customers as well".
Well, no.
Up until this point in the text we equated the development team to the company. By claiming that someone internal (be it operations, marketing or product management) is also a customer we are shifting our focus to the development team.
Let me say it clearly: As development team we don't have customers. We have employers and colleagues. Calling them "customers" might be useful when trying to convey the message that we provide a service to our colleagues and employers, but is otherwise causing confusion.  As employees we are getting a salary to promote the interests of our employer and that's it. I won't get a lower salary because my employer is unhappy with my work this month, and the contract between us have different obligations than those between a service provider and a customer. If I'm working towards my employer's interests it is time to face the cynical truth - most companies care about the customer only in terms of financial gain and a customer is nothing more than a walking bag of cash. Sure, it's very difficult to have a profitable product that does not align well with the customer's needs but when the two values are in conflict, I should be choosing the (long term) monetary1 business value over customer satisfaction.

So, Short version: Perceivable quality is not quality and it's important to remember that the customer is a (very important) mean to achieve our goals, but not the goal itself.

1 Thanks to Brent Jensen for reminding me that business value could be more than simply monetary. It's a bit harder to measure, and I believe it makes the point even more relevant - this business value could be the company's reputation, business contacts or even its moral values, for those fortunate enough to work in a company that cares about those


  1. Hi Amit, great post!

    I would like to state one important fact, context is the King!

    Your statement:
    "Perceivable quality is not quality and it's important to remember that the customer is a (very important) mean to achieve our goals, but not the goal itself."

    may be perfect feet for your company, but I worked for companies where Joel comment "The operations team are your customers as well". was really true.

    In those companies, customer is one who is paying for the quality. So phone os updates are not phone users quality. This update has set of quality improvements by phone maker, and that quality is paid by the company, not the phone user (because he has already paid for the phone).

    Question, do you have some kind of automation for writing multilingual posts?

    Regards, Karlo.

  2. Hi Karlo, thanks for the detailed response.
    I'm not sure I completely follow what you've said about the customer being the one paying for the quality, could you elaborate?

    As for automation for multilingual posts - I don't have one, that's one reason why it takes me a lot of time to write posts.
    I don't trust machine translation to be able to catch the nuances that exist in each languages (My Hebrew posts tend to be shorter, and I'm pretty sure that if translated word-by-word to English would be considered at least not politically-correct if not straight out rude). I also use the process of translating the post as a way to review what I've written, sometimes fixing and changing even entire passages.

  3. This is from the book from Gerald M. Weinberg book Agile Impressions, where customer and quality are explained:

    The relativity of quality.

    Quality is meeting requirements.

    Quality is meeting some person’s requirements.

    Every statement about quality is a statement about some persons(s).

    Who is the person behind the statement about quality? (heuristic for identifying quality.)

    More quality for one person may mean less quality for another.

    Quality is value to some person. (we usually hear this definition from testers. But it is just the conclusion, all previous quality quotes are essential part of this definition.)

    By value, I mean, “What are people willing to pay (do) to have their requirements met.”


    I hope this elaborates my statement that customer is one that is paying for the quality.

    Regards, Karlo.