Changing the numbers does not change the reality

I am a huge proponent of using analytics and other metrics to drive business decisions, but I repeatedly see people making a huge and avoidable mistake. Instead of using the data to determine the best strategy, they use data to justify their intuition. A good analyst can use data to draw virtually any conclusion and if the analyst is pushed in a certain direction by the business leader, all the data does is provide people with cover for the decision rather than leading you in the optimal direction.

The same situation applies to financial analysis. I have seen people frequently manipulate numbers, often with the approval or even encouragement of the target audience, to tell the story people want to hear. I have seen this manipulation in sales, in corp dev and in internal forecasting. In all situations, it is actually just a rationale to make a decision the person already wants to make.


Data manipulation

The first part of the problem is manipulating the data. I am not talking Enron here, but more subtly and maybe not even intentionally. People will often select the data that supports their position while discounting the other information. If you want to greenlight a certain feature, you may look at the impact on retention while neglecting the impact on monetization and rationalize it by saying it is a retention feature. Regardless of whether it is a retention or monetization, your goal is to optimize lifetime value (LTV) so you need to look at the data holistically. Continue reading “Changing the numbers does not change the reality”

How startups should use metrics

I recently came across a fantastic presentation on startup metrics by Andreas Klinger. It is embedded below but given its length I wanted to highlight the key takeaways:

  • The biggest risk for a startup is not failing to create a good product with a market; it is having a competitor come up with something a little better. Great example is Lyft, which I am sure is a little envious of Uber.
  • There are four stages for a startup to succeed. The first is discovery, generating the product idea. The second is validation, making sure the market wants the product. The third is efficiency, being able to supply the product cost effectively in quantity. Then there is scale, delivering the product to millions.
  • To look at it from the user perspective, there are two key elements: finding the product the market needs and then optimizing (the former encompassing discovery and validation, the latter representing efficiency and scale). To find a product the user needs, you need to understand these needs and create something that will be sticky (i.e., that they will return to) and viral (they will talk about). To optimize, you then need to build out the right revenue model and level, and then scale.
  • According to Klinger, 83 percent of startups are in the discovery phase (empathy, stickiness and virality) while most analytics are around revenue and scale.

    Andreas Klinger
  • A/B tests, funnels, referral optimization, etc., are about optimization, not innovation and cannot replace creating a great product that people want.
  • There is a way to get product insights from data to create that innovative product and you can do it with a much smaller number of users. They key is looking at whether people stay on your site or in your app, in other words, whether they are hooked.
  • Focusing on improving metrics creates a false positive, you can always improve ad conversions or funnels but what looks good for investors does not necessarily improve the product. You may be converting or funneling the wrong users.

Continue reading “How startups should use metrics”