Most of my posts about the reasons and methodology for creating accurate customer lifetime value (LTV) predictions have focused on the numbers and metrics, but a key element to predicting accurately LTV is observational (qualitative) data. It all comes down to more data is better, so predictions with qualitative data are going to be more accurate than those that rely solely on quantitative data. A mistake that is commonly made in the analytics world, and particularly in gaming, is to disregard anything that is not a quantitative value.
Some examples of incorporating effectively qualitative data
The example that had the most impact on me is that Billy Beane and the Oakland A’s, the subject of Moneyball (and multiple blog posts by me), has one of the highest scouting budgets in baseball. Scouts provide data on variables, like mental make-up and desire to win, that are not evident in the historical metrics. So although Beane makes personnel decisions based on metrics, he has also invested large sums in getting qualitative data (scouts watch players and prospects and then report on how they perceive the player’s skills). This approach has proven successful, as Beane’s A’s again surprised people by winning their division last year. What Beane has mastered is finding a way to incorporate the scouting reports with the available quantitative metrics.
As Nate Silver points out in his book The Signal and The Noise, “The organization [Oakland A’s] still very much believes in rigorous analysis. The rigor and discipline is applied, however, in the way the organization processes the information it collects, and not in declaring certain types of information off-limits.”
A second compelling example of the value of incorporating qualitative data is Google. Although Google captures and analyzes terabytes of data and runs over 6,000 A/B tests just on its search product annually (according to Hal Varian, Google’s Chief Economist), it also uses panels of human evaluators on a series of representative search queries. They then determine which statistical measurements are best correlated with these human judgments about importance and practicality. Given Google’s success (you probably used it for a search today), its strategy of integrating observational data with its quantitative data is hard to argue.
Problems with relying solely on quantitative data
The biggest mistake people run into when relying on quantitative data is assuming or ignoring causal relationships. You can always find patterns in data that explain historical results. By assuming that this relationship led to the result you may mistake coincidence for a behavior that can be exploited. For example, until several years ago there was a virtually perfect correlation between the winner of the Super Bowl and how the stock market performed (if the AFC won, the market would go one way, if the NFC won the market would go the other). If you based your investment decisions on this data, you would have lost everything in the last few years because the pattern broke. If you had incorporated qualitative data, however, by realizing the market and the Super Bowl winner had no relationship, you would of performed much better financially.
More formally, the above problem if taken further and used by even very smart analysts leads to the over-fitting of data. With big data, there are sometimes hundreds if not thousands of variables, and a smart analyst can almost always go back and create a formula integrating some of these metrics that explains the historical results. Many of those metrics, however, may not have actually influenced the result so if you accept the formula and put your resources into affecting those metrics, you actually will not have any impact on your performance.
How to generate qualitative data
- The best way to generate qualitative data is to play the game. It sounds very basic but it is almost a lost art with mobile and social game developers. Play—and keep playing—the game and see what you are experiencing, and how that fits with the quantitative data. If the retention and monetization numbers you are getting from a beta test are good, but you play the game and it obviously is not good, do not think you have a successful product. Although you are a sample of one, you can tell if the game is fun or not. If it is clearly not fun, it is not going to be successful. You may have incorporated well some techniques to enhance your numbers, but players are smart and they are not going to waste their time and money on a bad product.
- Focus test your game extensively. While you and your team can determine extreme cases, whether your game is awful or awesome, it is less likely that your team in itself can provide all the qualitative information you need. First, you may not be in the target demographic. Second, you are very close to the game. You understand how everything works and what the player is supposed to do. It is hard to find the “plot holes” because you have the backstory in your head. I like to bring in focus testers at all stages of a game’s development, from having them look at storyboards and concept arts to examining each key milestone. You will understand how your target market interacts with your product, what they really like and what you have to fix. This data can then supplement the metrics you get from larger beta or geo-locked testing.
- Survey your players. Sean Ellis, CEO of Qualaroo and first marketer at some great tech companies like Dropbox and LogMeIn, argues surveys are a great tool to determine what your player’s must-have experience is with your game. If there is no must-have experience, it also shows that you will not have strong retention numbers. By using surveys, you can understand what that experience is and whether people can live without your game (and what percentage can) and thus use this data to, among other things, improve the accuracy of your LTV by predicting better how long the player will play.
More data is better
The most compelling case for using qualitative data in your analysis, development and LTV predictions is the simplest: More data is better. As I mentioned earlier, predictions with qualitative data are going to be more accurate than those that rely solely on quantitative data. As long as you rigorously incorporate that data and do not let it turn into an excuse for ignoring the metrics, your development decisions and LTV predictions will be better.
The Business of Social Games and Casino 