The risks of market research

I have written multiple times recently about the perils of market research and a recent article by Kristen Berman and Dr. June John, [Don’t] Listen to Your Customer, builds on these perils. Berman and John write, “a sole reliance on customer input and feedback is built on an antiquated model of human decision making that assumes humans are rational,” and time again we see that people’s decision are often not driven by optimizing utility. Thus, talking to people and understanding their desires and aspirations to design products fails because most people do not know their desires and aspirations.

People do not answer accurately

The core problem with market research is you do not get true answers. In research done by Berman and John, they found that a small minority answered a question correctly about their own intent and importance. They asked people why they were or were not saving for retirement. 86 percent of people responded that saving for retirement was important. That response conflicted with the reality that in a typical company (without automatic opt-in), less than 50 percent of people would sign up for the retirement program. The authors write, “[t]he real reason people save for retirement or (don’t save) is because they’ve been defaulted into a savings plan. Companies who automatically enroll employees in a retirement savings plan increase savings rates by 50 percentage points. It has nothing to do with a preference for the future or not having enough money.”

Data is both inaccurate but also does not reflect motivators

Not only do people provide a response inconsistent with their actions, they often do not understand underlying causes of their behavior. In the retirement plan example, Berman and John followed up by asking why they had not enrolled in their company plan. Although data showed the key driver was whether it was or was not an opt-in plan, only 6.2 percent of respondents cited anything related to form design or default enrollment.

Basing decisions on these responses could have ruinous results. As the authors point out, “if we relied solely on data from these traditional qualitative and quantitative research methods, we would likely spend a lot of time and money building solutions that would miss the mark, and for sure be less effective than just designing a default enrollment.”

People are not lying

It is not that people are trying to mislead you when they respond inaccurately, the fundamental issue is that we are not conscious about the mental processes leading to decisions. We are also not cognizant that we do not know our own decision making processes.

Another study cited in the article showed over 70 percent of people did not know the underlying reason for a decision they made. Berman and John asked people if a default setting for organ donation and retirement savings would impact their choice. Despite data that shows such defaults nearly doubles the chance of someone opting in, 71 percent of people who said the default would not impact their retirement or organ-donation decision were “very” or “completely” confident in their answer.

The authors cite another study that reaffirms how confident people are in their incorrect responses. “They showed participants two female faces and asked which face was more attractive. After the participant pointed to a face, the researcher secretly swapped photos to the one they didn’t choose (think David Cooperfield). Then, the participant was asked to explain why they chose the face (which they didn’t actually pick). A majority of participants didn’t even notice the face swap. And furthermore, they went on to explain in great detail why they preferred that face. Most participants made up the reasons—good reasons—for liking a face that they didn’t actually choose. Only a third of people realized they had been tricked.”

Why it happens, ask Kayleigh McEnany

Given that people are not intentionally trying to be dishonest, you can ask why they are so inaccurate predicting and understanding their own behavior. Researchers have given this occurrence the name the Internal Press Secretary. Berman and John explain, “[t]he brain’s Press Secretary must, like a president or prime ministers’ press secretary, explain our actions to other people. “I don’t know” is simply not an acceptable answer. Many times the brain’s Press Secretary is correct, but other times, it’s just guessing, piecing together clues in order to give a plausible answer. The tricky thing is, we don’t know when our Press Secretary is using facts and when it’s just guessing.”


Don’t ask, test

Given the challenges people face understand and explaining the actions they will take, product managers and designers need to find alternatives to traditional market research. One of the most powerful is to test what people will react to different choices. This testing can take the form of an ABn or multi-armed bandit test, where you present the options to a test and control group. It can be a preference test where you show customer or potential customers different concepts and see which one they are most likely to purchase or would spend the most for.

You can also look at responses in similar circumstances. How did a similar product or service perform? Did competitors launch comparable features? I often find that looking at adjacent or even completely different industries helps understand people’s true preferences. The key is using one or multiple tools that show actual decision making rather than relying on what customers believe is their preference.

Key takeaways

  1. A sole reliance on customer input and feedback, traditional market research, is built on a model of human decision making that assumes humans are rational, while in practice we are not.
  2. Not only do people provide a response inconsistent with their actions, they often do not understand the underlying causes of their behavior.
  3. Use one or multiple tools that show actual decision making, such as ABn testing or looking at reactions to similar initiatives in adjacent industries, rather than relying on what customers believe is their preference.

Lifetime Value Part 30: Why clumpiness should be one of the KPIs you focus on

When calculating customer lifetime value (LTV),there is one KPI that is often neglected, clumpiness. Clumpiness is a term coined by Eric Bradlow, a marketing professor at Wharton. Think of clumpiness as binge-watching Ozark (or, if you must, Tiger King). Rather than an extended, constant period of consumption, a short, intense buying burst. By understanding and tracking this behavior, you have a significant weapon for improving LTV.

Defining clumpiness

Clumpiness refers to the fact that people buy in bursts and that those customers could be extremely valuable. When calculating customer value and segmentation, we focus on analysing recency, frequency and monetization of the customer (what Bradlow refers to as RFM). This analysis is based on customers making purchases in a regular pattern, i.e. coffee, diapers or milk. Bradlow’s analysis, however, shows that for certain products (and I would classify social and casino games here), customers actually monetize in bursts. Thus, you need to add C for clumpiness to your RFM modeling.

Why clumpiness is important

Bradlow researched how to predict the future value of customers (predicted LTV). Bradlow explains, “[l]et’s imagine you want … to predict who are going to be the valuable customers in the future. And you have four things you can use to predict it. As I mentioned: recency, frequency, monetary value and let’s say the marketing spend towards the customer. Those are the classic ways in which companies build what are called scoring models…. [T]he findings of my research suggest that higher clumpy customers are worth more out of sample, meaning in their future value, even after controlling for RFM and marketing expenditure — which means we have found another variable that firms should track [concerning] our customers and use it to predict their worth in the future.”

This finding is particularly interesting as most companies, especially in the game space, currently base their LTV projections on the RFM model. While this calculation may have worked in the traditional retail economy, consumption has evolved, especially for digital goods. Binge consumption is a fact of life in the entertainment space, and gaming sits squarely at the center of the modern entertainment environment. This analysis is consistent with Bradlow’s findings, where he says, “[i]f you look at historically purchased goods, clumpiness really isn’t there. But if you look in the new wave, the new economy, clumpiness is pervasive in every data set I’ve analyzed.”

Using clumpiness insights

Calculating clumpiness should be easy and not require tracking any new events. It is the same data you are using to calculate R, F, M and LTV. There are then several applications for this insight:

  • Incorporate it into your segmentation to get a better understanding of who your VIPs and high value players are, then focus your premium treatment (and benefits) on these players.
  • Use clumpiness to predict better what players are likely to become VIPs. This will help you concentrate your early retention efforts and reinvestment.
  • Focus reactivation on your clumpy players. Bradlow explains, “[i]f you reactivate them, they’ll come back and be clumpy again, and do a lot of stuff in the future. “
  • Do not simply rely on the KPIs you have always been trusting. Do not rely on simple theories of how your players behave, also look at things about arrival time or when people play. People who come in bursts, then go away and then come back in bursts and then go away are fundamentally different and have a different LTV.

By understanding clumpiness, you have a more accurate predictor of LTV. Once you know how to predict your LTV, you can then impact it by making changes that drive these variables.

Key takeaways

  1. We normally focus on analyzing recency, frequency and monetization of the customer but by adding a new KPI, clumpiness, we get a much better understanding of their expected value.
  2. Clumpiness refers to the fact that people buy in bursts and that those customers could be extremely valuable.
  3. Clumpiness can help you better segment players, predict VIPs and target your reactivation efforts and spend.