Requirements Satisfaction ≠ Customer Satisfaction

Ivan Jureta

There is engineering quality of a product or service, which is fitness to the specification, and there is perceived quality, or subjective quality, which is proportional to the distance between expectations and experience of the person asked. What is the relationship between these, between specification, requirements, expectations, and experience?

This is a longstanding question in requirements engineering [1,2] and software quality research [3,4]. 

One approach is to focus on the relationship between the specification and requirements. In that case, the classical view is the following: for some given constraints, the specification should satisfy requirements [1]. And that’s it. Satisfaction can mean different things (more on that here), but in all interpretations of satisfaction, the need is to demonstrate that if the conditions defined in the specification are true, and given constraints are true, then requirements statements will be true as well.

The satisfaction relationship can be operationalized, so to speak, by logical entailment (monotonic [1] or non-monotonic [2]), but it could be statistical correlation for instance, in case you are quantifying uncertainty about the specification’s implementation to satisfy requirements, and/or the uncertainty that you got the constraints right. The ways to operationalize the intuitive idea of satisfaction are complicated topics in their own right, and I leave them for another text.

Therefore, in the classical approach, because there is no explicit concept of customer satisfaction, it must be that requirements convey all that engineers need to know to make sure that, provided engineering quality is high, customer satisfaction is high. 

The important assumption, then, is that the level of requirements satisfaction correlates positively and strongly with customer satisfaction.

Customer satisfaction is evaluated usually by asking the customers directly, via less (e.g., net promoter score [5]) or more complex questions [6], or by taking a proxy, such as a measure of product or service usage over time. 

In which case can we say that the assumption (requirements satisfaction and customer satisfaction strongly and positively correlate) is true? If the evaluation of customer satisfaction indeed correlates positively and strongly with requirements satisfaction.

What if there is no such correlation? Why could that occur?

The obvious answer is that we satisfied the wrong requirements, or that we did not satisfy the right requirements enough, or some combination of the two. What could cause these cases?

One cause is that there were not enough resources (skill, time, etc.) to elicit the right requirements, and all the right requirements, to produce the specification that satisfies these, and then to make the product/service to specification, and take it to market.

Another is that the requirements believed to be the right ones changed, so that requirements at run time are different from those at design time. It may be the case that there is a contract which specifies that a specific set of requirements, considered at design time, needs to be satisfied – those taking on that obligation (more on requirements contracts here) will be producing a specification to satisfy those requirements, and may not be interested at all (which depends on how the contract is written) in run time requirements and differences between run time and design time requirements. While such a contract will manage some aspects of the relationship between those who provide requirements and those who design and make products/services to satisfy them, it is unlikely to constrain what customers think at design time and at run time. 

In short, if those who provide requirements are not customers, and they are free to change their requirements over time, there will be a risk that the assumption above does not hold. A typical case are business to consumer companies, where products and services are intended for customers, and there are those in companies – such as product designers and product managers – who are expected to provide requirements which, if satisfied, should result in high customer satisfaction; i.e., there are roles in the organization responsible for understanding the correlation between requirements satisfaction and customer satisfaction.

In the classical view in requirements engineering, it is safe to say that satisfaction of requirements is a function of the statements true of the specification, and statements true of constraints that the future. If S[R] is the overall level of requirements satisfaction, S are the properties of the specification, and K the constraints, then in the classical view, there is a function f_1 such that

S[R] = f_1(S,K)

If we now wanted to introduce customer satisfaction explicitly, several observations are important. 

  1. S[R] = f_1(S,K) ignores the implementation of the specification, so the S[R] = f_1(S,K) applies to design time, not run time. We rewrite it S[R,design_time] = f_1(S[design_time],K[design_time]).
  2. Customer satisfaction, let’s denote it S[C], can be measured only at run time, when customers can actually experience the product/service, within a given set of run time constraints K[run_time]; so we rewrite S[C] as S[C,run_time]. 
  3. The product/service is implemented according to the specification, and therefore, engineering quality matters: there can be a difference between what S states as properties of the product/service, and what the actual run time properties of the product/service are. Let P[run_time] denote the statements true of the customer’s experience of the product/service at run time.
  4. Customer satisfaction does not depend solely on the performance of the product/service. It depends also on customer expectations prior to experiencing the product/service at runtime. Let E[run_time] denote statements true of expectations at runtime.

Based on the above, we have these relationships:

S[R,design_time] = f_1(S[design_time],K[design_time])

S[C,run_time] = f_2(E[run_time],P[run_time],K[run_time])

P[run_time] = f_3(S[design_time])

It follows that the satisfaction of requirements at design time depends on assumptions about the future, and specifically about

  • run time relationship between the specification and its implementation, i.e., P[run_time] = f_3(S[design_time]);
  • run time relationship between customer satisfaction and customer’s expectations and experience of the product/service, and constraints within which product/service runs, i.e., S[C,run_time] = f_2(E[run_time],P[run_time],K[run_time]).

Laying these clearly expands the scope of the “requirements problem”, making it clear that when we produce specifications that should satisfy requirements, we do so while making many assumptions about run time, when customers will be able to experience the product/service, and we will know how good our predictions were.

References

  1. Zave, Pamela, and Michael Jackson. “Four dark corners of requirements engineering.” ACM transactions on Software Engineering and Methodology (TOSEM) 6.1 (1997): 1-30.
  2. Jureta, Ivan, John Mylopoulos, and Stephane Faulkner. “Revisiting the core ontology and problem in requirements engineering.” 2008 16th IEEE International Requirements Engineering Conference. IEEE, 2008.
  3. Kan, Stephen H. Metrics and models in software quality engineering. Addison-Wesley Professional, 2003.
  4. Kitchenham, Barbara, and Shari Lawrence Pfleeger. “Software quality: the elusive target [special issues section].” IEEE software 13.1 (1996): 12-21.
  5. Fisher, Nicholas I., and Raymond E. Kordupleski. “Good and bad market research: A critical review of Net Promoter Score.” Applied Stochastic Models in Business and Industry 35.1 (2019): 138-151.
  6. Hill, Nigel, and Jim Alexander. The handbook of customer satisfaction and loyalty measurement. Routledge, 2017.
  7. Parasuraman, Anantharanthan, Valarie A. Zeithaml, and Leonard L. Berry. “A conceptual model of service quality and its implications for future research.” Journal of marketing 49.4 (1985): 41-50.
  8. Zeithaml, Valarie A., et al. Delivering quality service: Balancing customer perceptions and expectations. Simon and Schuster, 1990.