I am a huge proponent of using analytics and other metrics to drive business decisions, but I repeatedly see people making a huge and avoidable mistake. Instead of using the data to determine the best strategy, they use data to justify their intuition. A good analyst can use data to draw virtually any conclusion and if the analyst is pushed in a certain direction by the business leader, all the data does is provide people with cover for the decision rather than leading you in the optimal direction.
The same situation applies to financial analysis. I have seen people frequently manipulate numbers, often with the approval or even encouragement of the target audience, to tell the story people want to hear. I have seen this manipulation in sales, in corp dev and in internal forecasting. In all situations, it is actually just a rationale to make a decision the person already wants to make.
Data manipulation
The first part of the problem is manipulating the data. I am not talking Enron here, but more subtly and maybe not even intentionally. People will often select the data that supports their position while discounting the other information. If you want to greenlight a certain feature, you may look at the impact on retention while neglecting the impact on monetization and rationalize it by saying it is a retention feature. Regardless of whether it is a retention or monetization, your goal is to optimize lifetime value (LTV) so you need to look at the data holistically.
Another way to manipulate data is to only look at certain cohorts. If you are deciding whether to invest further in a project, you may focus on your best users and make the argument that you scale that particular cohort you will make $500 million. The problem is you are looking at a small group that is not indicative of performance at scale. This type of manipulation is particularly prevalent when companies are pursuing investment or a sale. It overvalues the metrics the game will have at larger numbers and thus justifies a higher valuation.
Another area where you can manipulate data that is close to my heart is LTV. I have written many times how central LTV is to the success of the product and how it should drive user acquisition. Because of its importance, however, there is a large incentive to manipulate this metric. If you calculate a higher expected lifetime value, it justifies spending more on user acquisition. It could also mean the difference between keeping a game live and sun-setting it. As it is such a complex forecast, it is also easier to manipulate. If you change the expected life of a user, or the monetization curve, that would have a major impact on expected lifetime value (remember, lifetime value is a forecast, not a calculated result). So if an analyst is pressured to change the projection of how long a user will stay with a product, they can change the entire LTV projection.
There are many other areas where metrics can be manipulated, from virality (who is counted as a viral user versus an organic user) to retention (are you counting users separately who play the same app on multiple devices) and even monetization (are you using gross or net numbers, are you forgetting chargebacks, etc.) and the purpose of this post is not to provide a framework to manipulate data. The important issue is identifying ways that data is being manipulated, either intentionally or unintentionally, and then removing those biases so you can again let data drive optimal decision making.
Numbers manipulation
Analytics are not the only area open to manipulation, as spreadsheets and other financial reporting are frequently “adjusted” to provide the results people want rather than the information they need. Spreadsheets have an aura of objectivity but they are just as easily manipulated as a Word document. Sales projections, whether for investors or management, might increase by 10, 20 or 100 percent not because the business environment has changed but because they need to tell a different story.
In corporate development (especially M&A initiatives), where companies are risking millions or even billions of dollars, this same type of manipulation is common. If the selling company cannot make a good case for the sale, they adjust their projected numbers. This, however, is easily seen through and expected. The bigger problem is when the acquiring company really wants the deal to go through (either for personal reasons, ego or any number of external motivations) they too adjust the numbers to make the synergies look better (e.g., cost savings, sales improvements) until those synergies justify making the deal.
Just as I wrote with the issues of data manipulation, this should not be taken as an exhaustive list of manipulation of numbers (nor a roadmap) but as a warning of why spreadsheets should not be considered gospel. It is very easy to change a variable or value (sometimes by accident) that changes the entire financial story.
One of the biggest problems afflicting companies
These are issues that companies are dealing with daily. While virtually everyone acknowledges the strength of analytics and strong financial analysis, they are undercutting the value (or even destroying it) by using the numbers improperly. This puts them in the same position as companies that were not or do not use data.
Conclusion
At some point, you need to decide if you want to use metrics and financial analysis to create a great company or as cover to make decisions you want to make. You will only see the benefits of being a data driven company if you look at the numbers objectively and let those collecting the numbers use their best efforts to create accurate analysis. This is also a cultural issue, as even subtle pressure can impact the data and analysis.
As mentioned earlier, it is not always a conscious decision. You may gravitate to the data or analysis that supports your position or initiative while discounting other data. Unfortunately, this course is just as damaging as you may make sub-optimal decisions. The best way to avoid this problem is by deciding a priori what data you will look at to make the decision and what metrics you need to see to move one way or the other.
Key takeaways
- One of the biggest issues facing virtually all companies is the manipulation of data and numbers (either subconsciously or intentionally), offsetting the ability to make optimal data-driven decisions.
- With analytics, data can be misused by selectively choosing which data to use.
- In financial analysis, decisions are often made based on spreadsheets and analysis that can be made to tell any story by “adjusting” the key variables.
The Business of Social Games and Casino 
One thought on “Changing the numbers does not change the reality”