Category: Analytics

The use of analytics in the marketing, development or deployment of social games

Word-of-machine effect with recommendation engines

Recommendation engines have been around for years, it is one of the early drivers of Amazon.com’s success (first written about in 2003), but recently they have evolved from a competitive advantage to a cost of doing business. A recent article in the Harvard Business Review, When Do We Trust AI’s Recommendations More Than People’s? by Chiara Longoni and Luca Cian, provides useful advice for leveraging further AI-driven recommendation engines.

Amazon rec engine

The central finding by Longoni and Cian is that accuracy is not the only element important in recommendation engines, but also context. They write, “simply offering AI assistance won’t necessarily lead to more successful transactions. In fact, there are cases when AI’s suggestions and recommendations are helpful and cases when they might be detrimental. When do consumers trust the word of a machine, and when do they resist it?”

Word-of-machine effect drives acceptance of AI recommendations

For recommendations to impact positively the customer experience, a key factor is what is being recommended. Longoni and Cian found that when customers focus on the functional and practical aspects of a product, its utilitarian value, they trust a machine. If they focus on the experiential and sensory aspects of a product, its hedonic value, they prefer and trust a human recommendation. If a customer is focused on utilitarian and functional qualities, then, from a marketer’s perspective, the word of a machine is more effective than the word of human recommenders. For someone focused on experiential and sensory qualities, human recommenders are more influential.

This phenomenon is referred to as word-of-machine effect. The authors write, “the word-of-machine effect stems from a widespread belief that AI systems are more competent than humans in dispensing advice when utilitarian qualities are desired and are less competent when the hedonic qualities are desired. Importantly, the word-of-machine effect is based on a lay belief that does not necessarily correspond to the reality.… The fact of the matter is humans are not necessarily less competent than AI at assessing and evaluating utilitarian attributes. Vice versa, AI is not necessarily less competent than humans at assessing and evaluating hedonic attributes.”

The word-of-machine effect extends to how people evaluate a product. If a product, like food (or a slot machine), that is evaluated based on hedonistic traits, is recommended by AI rather than a person, the consumer is more likely to discount its quality after trying it. In a study cited by Longoni and Cian, “[they] recruited 144 participants from the University of Virginia campus and informed them that we were testing chocolate-cake recipes for a local bakery. The participants were given two options: one cake created with ingredients selected by an AI chocolatier and one created with ingredients selected by a human chocolatier. Participants were then asked to eat one of the two cakes, which were identical in appearance and ingredients, and rate the cake for two experiential/sensory features (indulgent taste and aromas, pleasantness to the senses) and two utilitarian/functional attributes (beneficial chemical properties and healthiness). Participants rated the AI-recommended cake as less tasty but healthier than the cake recommended by the human chocolatier.”

Customers, however, trust AI-generated recommendations more when utilitarian features are most important. In another study, when consumers who were looking to purchase a winter coat were given recommendations based on practical features (warmth, breathability, etc), they preferred an AI shopping assistant over a human, and the more they cared about hedonic features, the more they preferred a human shopping assistant over an AI.

Augmented intelligence

Another interesting finding by Longoni and Cian is that consumers will embrace AI recommendations if they believe a human was part of the recommendation process. In one experiment, the authors framed AI as augmented intelligence that enhances and supports human recommenders rather than replacing them. The AI-human hybrid recommender fared as well as the human-only recommender even when experiential and sensory considerations were important. Going back to Amazon, they are quite adept at combining human and AI recommendations. As part of their recommendation process, they will show customers what other customers like them have bought, thus it is effectively the other customers, not AI, making the recommendation.

Human recommendations

Balancing word-of-machine effect

One way to mitigate word-of-machine effect is telling customers about the phenomenon (though wording it simpler than an academic would). Longoni and Cian found that, by prompting people to consider a different viewpoint about the recommender’s ability, it reduced the word-of-machine impact.

Another option that the authors found useful is using a chat bot. In one study, “[they] invited 299 online respondents to read about an app called ‘Cucina’ that would rely on AI to give recipe recommendations. Within the app, the participants were able to interact with a chat bot – an AI chef – that was programmed to assist them. The chat bot greeted each participant and introduced itself (‘Hi, Mark! I am here to suggest a recipe for you to try!’). The AI chef then delivered the consider-the-opposite protocol using a fun, interactive nudge: ‘Some people might think that an artificial intelligence chef is not competent to give food suggestions, but this is a misjudgment. For a moment, set aside your expectations about me. When it comes to making food suggestions, could you consider the idea that I could be good at things you do not expect me to be good at?’ That resulted in more favorable perceptions of the AI recommendation even when people were considering the experiential and sensory qualities of a recipe like taste and aromas.”

How to benefit from word-of-machine effect

As I mentioned earlier, effective AI recommendation engines have gone from being a competitive advantage to a must have feature of any online product. To provide the best customer experience, you need to understand the word-of-machine effect and biases it creates. It is important to remember these are biases, the AI can often give a better recommendation even if customers believe otherwise. Thus, you need to find a way to lead your customer to their best possible experience, without misleading them or creating the perception that it is a sub-optimal experience. The key is framing the recommendation in a way the customer will trust it, given what is being recommended.

Key takeaways

  • AI driven recommendation engines, popularized on Amazon and Netflix, have gone from being a competitive advantage in online and mobile apps to part of the cost of doing business.
  • While AI recommendations are often closer to what customers want, people trust human recommendations more when it is based on hedonic or experiential factors, and trust machines more when it is based on utilitarian factors; this is referred to as word-of-machine effect.
  • When presenting recommendations, you must be cognizant of word-of-machine effect; for recommendations around hedonistic properties try to bring in a human element (augmented intelligence) or explain that these are better recommendations.

Adjacent User Theory Shows How to Supercharge Your Game’s Growth

We are all constantly looking at ways to grow our game or app like Facebook or Twitter or Fortnite, so where better to start looking than Instagram. An article by Bengaly Kaba, The Adjacent User Theory, shows the methodology he used at Instagram to reignite its mega-growth. Kaba joined Instragram in 2016 and was instrumental in helping it grow from about 400 million users to over 1 billion.

Kaba credits his success to Adjacent User Theory, where potential users (adjacent users) are aware of the product, maybe tried it, but are not engaged customers. The app’s positioning or too many barriers in the early user experience often drive the lack of traction with these customers. In Kaba’s situation, the 400 million active users of Instagram represented instances where they found product-market fit but also missed over 600 million potential customers who either did not understand Instagram or how it would fit into their lives. According to Kaba, “our insight was that it is critical for growth teams to be continually defining who the adjacent user is, to understand why they are struggling, to build empathy for the adjacent user, and ultimately to solve their problems.”

Solving for the adjacent user

If you have a successful game or product, you almost certainly have a good understanding of your customers (or else you would not be successful). Essentially by definition, you do not have the same understanding of adjacent or future users, or else they would already be customers. Also, your future audience evolves over time, so what adjacent users are not getting now from your game might be different from what keeps your product appealing outside its core in six or twelve months.

To solve for the adjacent users, either your product team or a satellite of your product team needs to focus on these potential customers. Kaba writes, “[w]ithout a team dedicated to understanding, advocating, and building for your next set of users, you end up never expanding your audience. This stalls growth, and the product never reaches the level you aspire it to…. You can think about your product as a series of circles. Each of these circles is defined by the primary user states that someone could be in. For example Power, Core, Casual, Signed Up, Visitor. Each one of these circles have users that are “in orbit” around it. These users have an equal or greater chance they drift off into space rather than crossing the threshold to the next state. There is something preventing them from getting over the hump and transitioning into the next state. These are your adjacent users and the goal is to identify who they are and understand their reasons struggling to adopt. As you solve for them, you push the edge of the circle out to capture more of that audience and grow.”

Slack represents a good example of the power of adjacent users, according to Kaba. Slack has five user states — not signed up, signed up, casual, core free and monetized — and will lose potential customers at the transition between each of these five states. By understanding why people drop off between states before becoming monetized, Slack can make product changes that moves these adjacent users from non-customers to engaged customers.

Why companies are not focusing on adjacent users

While it seems obvious that converting adjacent users is critical for growth, many companies miss this opportunity. Kaba identifies three reasons for this problem

  1. Focusing on power users. It is natural to focus your development efforts on your VIPs, especially in gaming where a very small percent of your players drive most of the revenue. Thus, product development is geared to giving these players more of what they want, rather than bringing more customers into the tent.
  2. Personas. Many companies build personas (a fictional character created to represent a player type that might use a game) to determine who they are solving for. The mistake with this approach is that personas are usually created based on existing players, do not change as customers evolve, not based on relative usage/contribution and are too broad to be actionable.
  3. The home run swing. Product teams often look for the bold beat feature that will have a dramatic impact rather than a series of changes. Kaba writes, “they get bogged down by trying to establish product-market fit for a new set of users and never fulfill the potential of their current product-market fit.”

Slide1

How to solve for adjacent users

There are several ways to mitigate the above issues and appeal to adjacent users. To overcome the bias of VIPs, product teams need to cross a “cognitive threshold” and understand the experience of non-VIP customers as well as non-customers. This includes your team putting aside their personal biases if they are using the product, they are not building it for themselves but building for adjacent users.

To avoid having your product team focus on the huge wins, Kaba writes that they must “[r]emember…adjacent users are the users who are struggling to adopt your product today. Non-adjacent users could literally be everyone else in the entire world. Sure, non-adjacent users might be a larger market, but the barriers to their adoption are also dramatically higher. Companies that try to go too big too soon and often, skip the next obvious steps and fail to solve their current adoption problems.”

Identifying and defining your adjacent users

Once you understand the value of adjacent users and how to build for them, you need to find them. You need to look at cohort decay, keeping in mind the different circles or user states for your game. You need to look at these variables (ie. registering to purchasing) and identify the decline in each cohort. That declines represents the adjacent users.

You then need to build a theory on who they are and why they are struggling or not converting. You also need to realize you will not have perfect visibility into the answer. Kaba describes the process, “is to lay out multiple hypotheses of who the adjacent users are, choose which one to focus on strategically, force your team to look at the product through their lens, experiment and talk to customers to validate and learn, then update the landscape to make your next choice. I like to think about it as a snowball. You know very little at first, but as the snowball turns you collect more information about the adjacent user, which helps you collect more snow (users).”

It is also important to gain a deep understanding of your current users. If you find that your current users are predominantly, male, 35-50 years old and from the northeast in the US, you can then look at each of these attributes to find adjacent users. For example, maybe you are losing women. Maybe you are not appealing to 50+. Maybe users from the Midwest are not connecting.

Once you have hypothesis on your current users you can create multiple hypothesis of adjacent users. Kaba recommends, “starting with a bottoms-up analysis of your data. You do not need to spend weeks talking to users to get a sense for who your adjacent user is. Look at what is happening on the edges of these states in the data. The data will help you identify places in the product that people are dropping off. This is the starting point to help you develop hypotheses about why different segments of users are dropping off.”

Understand the why

Once you have identified your adjacent users, you need to understand why they are dropping off. To achieve this knowledge, your product team needs to empathize with the adjacent users. This is usually a challenge as they often think like your VIPs and current users and find it difficult to put themselves in the heads of people who do not like their game or product. There are three ways, though, to overcome this problem:

  1. Be the adjacent user. Your product team needs to be experiencing your game in the conditions and settings that the adjacent user is experiencing. Some examples are new user flows, empty states, and product states that require different levels of play.
  2. Watch the adjacent user. This is done by usability testing. It is best to do this in the adjacent users environment, as focus groups and in-office usage creates artificial conditions.
  3. Talk to the adjacent user. Once you have identified adjacent users, ask them why they are trying to use your product, what jobs they are trying to solve, and what other games they play or have tried.

Prioritizing adjacent users

Once you understand your adjacent users and have identified what they need from your game or product, you need to prioritize. You cannot solve all of their issues at once, instead you need to focus on creating the most long-term value for your company. You want to do it in the correct order or sequence so that your efforts build on each other.

First, the adjacent users you should approach are ones different on only one or two attributes. You will only have to make limited changes or additions to your product to appeal to them.

Second, they should align with your strategy. Will the changes you have to make to appeal to this segment help you achieve your long-term strategy and vision for the product. You cannot please everybody always and trying to do so could potentially take away from what is making your game successful.

Third, start with adjacent users already in your funnel, customers you are losing, rather than trying to appeal to a completely new segment.

Not a one time exercise

Once you have identified and made product improvements to appeal to your adjacent users, do not proclaim mission accomplished, as the pool of adjacent users is constantly evolving. Efforts to build for one segment will identify new segments. These efforts will also bring in new adjacent users who you can then convert. Finally, these efforts will create a new or enhanced value proposition, which means that both current and adjacent users will have a new equation in deciding whether to engage. You will then want to adjust based on how they react.

Key takeaways

  • Adjacent users represent a great opportunity for growth. These are potential players (adjacent users) are aware of the product, maybe tried it, but are not engaged customers
  • You solve for these players by looking at different states of your product (i.e. registration, play, purchase) and seeing who drops off at each of these states, then understand why these potential customers are dropping off.
  • You can make your product attractive by putting yourself in the place of the adjacent user, watching them use your game or product and talking to them.

3 Words I Hate

Last year I wrote about one of the most insidious phrases in business (which, ironically, became a talking point in US politics last month), and there are three other words that exasperate me when used by gaming companies. These words — gamification, whales and directional — often drive the wrong actions, ideas or initiatives.

Slide1

Gamification

Trying to gamify a game is the height of absurdity, or at a minimum shows you have not done your job well. A game, by definition, is a game, so why would you want to add gamification. If done properly, the product already will entertain customers. A successful product will have a strong core game loop, that will drive your players enjoyment, and thus retention and engagement. The core loop is a chain of actions that the player does over and over again.

Gamification becomes a problem as it is often used as a solution for a poorly designed game. Rather than creating a strong core loop that retains players, companies try to use tricks (gamification) to overcome the shortcomings of the product. Gamification is often a euphemism for adding features that bandage over underlying problems with the product.

This problem also holds for casino and social casino games. Slots, poker, bingo, etc., represent great core game loops that have been entertaining people for hundreds of years. You do not gamify a slot machine, its core game loop is already compelling. Poker is a fantastic game that people will spend a lifetime playing. These are already great games and cannot be gamified.

If you are trying to make necessary actions outside your core game loop (registration, purchases, CS, etc.) better for the customer, you are not actually making them into a game (gamifying) but looking at principles of consumer behavior and behavioral economics to make users more likely to complete the tasks. You are not going to create a registration process that is more fun than Clash of Clans or more entertaining than the most recent Disney movie, and that should not be your target. Instead, focus on making the task effortless, so the player returns to your (entertaining) game.

The argument against gamification is not a criticism of building strong meta-features that enhance the core game loop. Adding a progression system or social features build on the core game loop but you are not gamifying your game, you are putting it in a superior package.

Whales

While gamification is misleading and often used as an excuse, the word whales is insulting and creates the wrong approach to your best players. Many companies, particularly in the iGaming and social casino world, use the phrase whales to describe their most valuable players. If you just look at my last sentence, you should see what is wrong with that approach. These companies are using a derogatory and insulting phrase to talk about the customers who in many (if not most) social games drive the majority or revenue. Rather than a condescending phrase, we should treat these players with the respect they have earned. For those who would argue whales is not derogatory, describe your partner that way to them and see how well it goes.

The damage in using the phrase whales is more than semantics, as it creates the wrong approach to your best customers. When you look at a group of customers as big, fat animals (no offense to actual whales, who are beautiful creatures), you are likely to treat them in a condescending or exploitative manner. Having started and built two successful VIP programs, I have seen that on a tactical level, this is a bad strategy because your highly valued customers are then are put into conflict with the company. The VIPs are trying to optimize their experience; you are trying to get as much wallet share as possible. Long term, the VIPs are more likely to go to another game where they feel respected (just as you would leave McDonalds if you get a condescending sales person and go to Panera). If you want VIPs to stay, call them VIPs and treat them that way.

Directional

While not as insidious as whales, another dangerous word is directional. This phrase is often used during or after an AB test, when the results are not statistically significant. Even without significance, you accept the winning variant as a preferred solution.

The problem with looking at directional results is that there is a lot of noise, and you are likely basing your decision on luck, not on numbers. Statistical noise is the random irregularity we find in any real life data. They have no pattern. One minute your readings might be too small. The next they might be too large. These errors are usually unavoidable and unpredictable.

Using directional results is no better than making decisions without numbers, and can lead to the same consequences. If the consequences are minor, or you know you will pursue a certain strategy regardless, then the AB test is a waste of resources and you should not have deployed it (which is fine, not everything needs to be tested).

Using directional data, additionally, creates the illusion of a data-driven decision and is subject to confirmation bias, as people use directional data to support decisions they have already made. I have never heard the phrase directional used when the result is not what the product manager or GM was hoping for. In those cases, the results would be considered inconclusive. People use directional to justify a course of action where there is no real data.

Key takeaways

  • Game companies should avoid the phrase gamification, as their product is already a game and should be entertaining.
  • Whales is an insidious phrase as it describes your best customers in a derogatory way, potentially leading to treating them in a way that destroys long-term value.
  • It is misleading to use data that is directional, without statistical significance as relying on this data is like not using data at all.

The risks of market research

I have written multiple times recently about the perils of market research and a recent article by Kristen Berman and Dr. June John, [Don’t] Listen to Your Customer, builds on these perils. Berman and John write, “a sole reliance on customer input and feedback is built on an antiquated model of human decision making that assumes humans are rational,” and time again we see that people’s decision are often not driven by optimizing utility. Thus, talking to people and understanding their desires and aspirations to design products fails because most people do not know their desires and aspirations.

People do not answer accurately

The core problem with market research is you do not get true answers. In research done by Berman and John, they found that a small minority answered a question correctly about their own intent and importance. They asked people why they were or were not saving for retirement. 86 percent of people responded that saving for retirement was important. That response conflicted with the reality that in a typical company (without automatic opt-in), less than 50 percent of people would sign up for the retirement program. The authors write, “[t]he real reason people save for retirement or (don’t save) is because they’ve been defaulted into a savings plan. Companies who automatically enroll employees in a retirement savings plan increase savings rates by 50 percentage points. It has nothing to do with a preference for the future or not having enough money.”

Data is both inaccurate but also does not reflect motivators

Not only do people provide a response inconsistent with their actions, they often do not understand underlying causes of their behavior. In the retirement plan example, Berman and John followed up by asking why they had not enrolled in their company plan. Although data showed the key driver was whether it was or was not an opt-in plan, only 6.2 percent of respondents cited anything related to form design or default enrollment.

Basing decisions on these responses could have ruinous results. As the authors point out, “if we relied solely on data from these traditional qualitative and quantitative research methods, we would likely spend a lot of time and money building solutions that would miss the mark, and for sure be less effective than just designing a default enrollment.”

People are not lying

It is not that people are trying to mislead you when they respond inaccurately, the fundamental issue is that we are not conscious about the mental processes leading to decisions. We are also not cognizant that we do not know our own decision making processes.

Another study cited in the article showed over 70 percent of people did not know the underlying reason for a decision they made. Berman and John asked people if a default setting for organ donation and retirement savings would impact their choice. Despite data that shows such defaults nearly doubles the chance of someone opting in, 71 percent of people who said the default would not impact their retirement or organ-donation decision were “very” or “completely” confident in their answer.

The authors cite another study that reaffirms how confident people are in their incorrect responses. “They showed participants two female faces and asked which face was more attractive. After the participant pointed to a face, the researcher secretly swapped photos to the one they didn’t choose (think David Cooperfield). Then, the participant was asked to explain why they chose the face (which they didn’t actually pick). A majority of participants didn’t even notice the face swap. And furthermore, they went on to explain in great detail why they preferred that face. Most participants made up the reasons—good reasons—for liking a face that they didn’t actually choose. Only a third of people realized they had been tricked.”

Why it happens, ask Kayleigh McEnany

Given that people are not intentionally trying to be dishonest, you can ask why they are so inaccurate predicting and understanding their own behavior. Researchers have given this occurrence the name the Internal Press Secretary. Berman and John explain, “[t]he brain’s Press Secretary must, like a president or prime ministers’ press secretary, explain our actions to other people. “I don’t know” is simply not an acceptable answer. Many times the brain’s Press Secretary is correct, but other times, it’s just guessing, piecing together clues in order to give a plausible answer. The tricky thing is, we don’t know when our Press Secretary is using facts and when it’s just guessing.”

Kayleigh

Don’t ask, test

Given the challenges people face understand and explaining the actions they will take, product managers and designers need to find alternatives to traditional market research. One of the most powerful is to test what people will react to different choices. This testing can take the form of an ABn or multi-armed bandit test, where you present the options to a test and control group. It can be a preference test where you show customer or potential customers different concepts and see which one they are most likely to purchase or would spend the most for.

You can also look at responses in similar circumstances. How did a similar product or service perform? Did competitors launch comparable features? I often find that looking at adjacent or even completely different industries helps understand people’s true preferences. The key is using one or multiple tools that show actual decision making rather than relying on what customers believe is their preference.

Key takeaways

  1. A sole reliance on customer input and feedback, traditional market research, is built on a model of human decision making that assumes humans are rational, while in practice we are not.
  2. Not only do people provide a response inconsistent with their actions, they often do not understand the underlying causes of their behavior.
  3. Use one or multiple tools that show actual decision making, such as ABn testing or looking at reactions to similar initiatives in adjacent industries, rather than relying on what customers believe is their preference.
%d bloggers like this: