In the mobile gaming space, being data driven is largely synonymous with using quantitative data, with qualitative data (such as user research) relegated to third class status. Most game companies believe their decisions should be based on user activity in their product. They often act and allocate resources only to analyze their existing data, with design thinking and user research considered old fashioned and not as accurate. These companies, however, are missing an opportunity.
As I have said before, all data should be used to make decisions and optimize your product. Gameplay data is extremely valuable but so are qualitative data sources. Young children learn quite early that when you add a positive number to a number, the new number is larger. Thus, there is no way adding qualitative data to your quantitative data can make your analysis weaker. As long as the qualitative research does not detract from the analysis, it makes you stronger.
There is a great article, Qual vs Quant: when to listen and when to measure by Laura Klein, that explains when qualitative research is most useful. The article explains that you should constantly use both qualitative and quantitative data, but different situations require different focus.
When to focus on qual research
Qualitative research is important to understand why you are experiencing a problem or situation. Quantitative research tells you that you have a problem and what the problem is (i.e. D1 retention is down 20%).
What method in which situation
Klein highlights three different high level scenarios and then discusses whether qualitative or quantitative research would help most. In the first scenario, you are only considering changing one variable. You may be deliberating whether to change the maximum bet on slots in your casino. In this situation, quantitative analysis is optimal. With a change this small, users in a testing session or discussion will not give you any actionable information. Qualitative feedback will not provide failure to compensate for the time and money it takes to set up interviews, talk to users, and analyze the data.
The second scenario is when you are planning a multi-variable or flow change. You may be planning to implement a new feature, like a progression system, that would among other things require you adjust the user interface and journey. While you could launch it and measure performance, you would never know why it failed or succeeded. Qualitative research allows you to learn before launching what elements of the feature are compelling, what is confusing and how you can optimize the feature before you launch the feature.
The third scenario is prioritizing your roadmap. In this situation, a combination of qualitative and quantitative research is optimal. As Klein writes
The key here is that you want to look at what your users are currently doing with your product and what they aren’t doing with it, and you should do that with both qualitative and quantitative data.
Qualitative Approaches:
- Watch users with your product on a regular basis. See where they struggle, where they seem disappointed, or where they complain that they can’t do what they want. Those will all give you ideas for iterating on current features or adding new ones.
- Talk to people who have stopped using your product. Find out what they thought they’d be getting when they started using it and why they stopped.
- Watch new users with your product and ask them what they expected from the first 15 minutes using the product. If this doesn’t match what your product actually delivers, either fix the product or fix the first time user experience so that you’re fulfilling users’ expectations.
Quantitative Approaches:
- Look at the features that are currently getting the most use by the highest value customers. Try to figure out if there’s a pattern there and then test other features that fit that pattern.
- Try a “fake” test by adding a button or navigation element that represents the feature you’re thinking of adding, and then measure how many people actually click on it. Instead of implementing an entire system for making friends on your site, just add a button that allows people to Add a Friend, and then let them know that the feature isn’t quite ready yet while you tally up the percentage of people who are pressing the button.
What to do
There are some situations where you should rely on the quantative research, other situations where qualitative research is best, but generally a combination of the two is the optimal way to formulate your product strategy. By using both, you are collecting all available information and using this data to drive optimal decisions.
Key takeaways
- Qualitative research (survey, user panels, etc.) is often neglected by game companies who prefer to use quantitative data (KPIs) to drive decisions, but this approach neglects that more information is always better.
- In situations where you are only impacting one variable, quantitative data is the answer; if you are adjusting flow or multiple variable, lean on qualitative data; and if you are prioritizing your roadmap, use a robust combination.
- There are some situations where you should rely on the quants, other situations where qualitative research, but generally a combination of the two is the best way to formulate your product strategy.
