Eff the user

גרסה עברית

This idea has been running in my head for a while, and every so often I'll stumble upon a reminder to write something about it, and then have other things to occupy me. The most recent trigger was a talk by Joel Montvelisky in a local SIGiST conference where he spoke about the changing role of the testers. The subject is my objection to one of the modern testing principles.  I'll admit - I really like them. That is, with the exception of principle #5: "We believe that the customer is the only one capable to judge and evaluate the quality of our product".

I don't like this one a tiny single bit. I think it's a false statement and it drives wrong team behaviour and goal displacement. Worse, this idea seems to be ubiquitous in the industry, or at least wildly spread, as some of the variants of similar ideas out there show. I think that #5 is one of the most carefully worded expressions of this idea (as one cannot dismiss it with "a tester can't be a customer proxy\champion\advocate") So I'll stick with it when thinking through my disagreement. 

On the face of it, the principle is very solid - if we accept Jerry Weinberg's notion of quality being value to some person (and I like Bach's addendum to it), it only makes sense that the person who is getting the value is the best judge of it. 

Except when they aren't.

I am not claiming here that a product that is not used or bought by any of its target audience can claim to be "of high quality", but simply that there are cases where evaluating the quality of a product is something that the customer is not capable of doing. 

Imagine that you are getting an OS update to your phone. It improves memory management, fixes a security flaw and provides better logging in cases of a crash. Are you in any position to evaluate any of this? Would an average phone user be in such position? All of it brings value - better logging means faster debugging cycles and faster hotfix releases, a security update reduces the likelihood of the phone being hacked and the memory management improvement will mean you could run more applications in parallel and avoid lagging. But how do you even notice any of those? You don't see the crash reports (and there are other ways to improve response time), the security update is like an insurance - you only notice it if something bad happens, and  your phone has enough memory that you didn't notice any problem with the previous memory management scheme. In all of those examples, there are simply too many layers of indirection between the actual property and the perceivable result and the quality of the product, if assessed by the customer, must be assessed by a proxy measurement. In some cases, the customer might notice if you did a shitty job, but the distinction between "barely good enough" and "excellent" is not visible. 

Then there's the ambiguity about "who is the customer". Consider a bug fix that eliminates a scenario where the user is charged for only 3/4 of the actual use. By fixing it, we are taking "value" from the product's user, but adding value to the vendor. While such condition is extreme, there are more examples when one looks on B2B products - The full disk encryption scheme used by our IT department makes our computers freeze for a second every now and then, but provides the decision makers (who don't use their PC as intensely as other employees do) the peace of mind knowing that data could not be leaked by simply stealing a PC. Customer value? high. User value - not so much.

Now, to goal displacement and wrong team behaviour. A simplistic understanding of this principle drives a lot of focused into perceived quality and requires advanced logic-fu to justify any kind of improvement that isn't a new feature. Take for example an architecture improvement for enabling our product to scale to need. We can go with the current architecture for a year or so, and probably throw more hardware on it for the next five years, so, no real benefit to the user, let's not do that. The rule-fu required would be something along these lines:  "If we don't do it then we can probably patch the system for 5 years but it will gradually slow us down as we deal with issues as they appear, and will also be exposing customers to issues caused by this load. In addition, in five years time we'll have more components relying on the old architecture we want to replace, so the change will take longer and be more risky and complex". This sort of logic is not trivial and far from immediate, if we train our instincts to zero in on the customer satisfaction we'll have a tough time coming up with such scenarios, let alone justify such future-looking actions in face of a well trained gut-feeling. Most teams don't need another thing applying pressure and bias towards perceivable features. True, if we insist, everything we do has an ultimate impact on the customer, but sometimes the impact of good engineering practices is several times removed from the actual impact, and it will usually be only one of the factors affecting that specific property of customer value. Imagine two similar SaaS products, both have the same capabilities, a similar size of customers and are distinguishable only by minor features that might appeal more to someone's personal taste. One product is built by a team of 10 highly skilled engineers that are running a tight ship. The 2nd is maintained by 100 people who work frantically to keep things afloat. The cost of operation for the two products is similar - The top notch engineers are getting payed 10 times as the less skilled ones, and the robust hardware and hosting services used by the first team are as expensive as the cost of throwing cheaper hardware  to gap for design problems in the second team. The perceived quality is the same, both companies can scale at roughly the same pace with roughly the same resources - is the quality of the two products equal? They are supplying the same customer value in different means, only the 2nd team is making it really difficult for their operations team.
I got a response for similar claim from Joel who said "The operations team are your customers as well".
Well, no.
Up until this point in the text we equated the development team to the company. By claiming that someone internal (be it operations, marketing or product management) is also a customer we are shifting our focus to the development team.
Let me say it clearly: As development team we don't have customers. We have employers and colleagues. Calling them "customers" might be useful when trying to convey the message that we provide a service to our colleagues and employers, but is otherwise causing confusion.  As employees we are getting a salary to promote the interests of our employer and that's it. I won't get a lower salary because my employer is unhappy with my work this month, and the contract between us have different obligations than those between a service provider and a customer. If I'm working towards my employer's interests it is time to face the cynical truth - most companies care about the customer only in terms of financial gain and a customer is nothing more than a walking bag of cash. Sure, it's very difficult to have a profitable product that does not align well with the customer's needs but when the two values are in conflict, I should be choosing the (long term) monetary¹ business value over customer satisfaction.

So, Short version: Perceivable quality is not quality and it's important to remember that the customer is a (very important) mean to achieve our goals, but not the goal itself.

---

1 Thanks to Brent Jensen for reminding me that business value could be more than simply monetary. It's a bit harder to measure, and I believe it makes the point even more relevant - this business value could be the company's reputation, business contacts or even its moral values, for those fortunate enough to work in a company that cares about those^↩