How startups should use metrics

I recently came across a fantastic presentation on startup metrics by Andreas Klinger. It is embedded below but given its length I wanted to highlight the key takeaways:

• The biggest risk for a startup is not failing to create a good product with a market; it is having a competitor come up with something a little better. Great example is Lyft, which I am sure is a little envious of Uber.
• There are four stages for a startup to succeed. The first is discovery, generating the product idea. The second is validation, making sure the market wants the product. The third is efficiency, being able to supply the product cost effectively in quantity. Then there is scale, delivering the product to millions.
• To look at it from the user perspective, there are two key elements: finding the product the market needs and then optimizing (the former encompassing discovery and validation, the latter representing efficiency and scale). To find a product the user needs, you need to understand these needs and create something that will be sticky (i.e., that they will return to) and viral (they will talk about). To optimize, you then need to build out the right revenue model and level, and then scale.
• According to Klinger, 83 percent of startups are in the discovery phase (empathy, stickiness and virality) while most analytics are around revenue and scale.

  

• A/B tests, funnels, referral optimization, etc., are about optimization, not innovation and cannot replace creating a great product that people want.
• There is a way to get product insights from data to create that innovative product and you can do it with a much smaller number of users. They key is looking at whether people stay on your site or in your app, in other words, whether they are hooked.
• Focusing on improving metrics creates a false positive, you can always improve ad conversions or funnels but what looks good for investors does not necessarily improve the product. You may be converting or funneling the wrong users.

• As I have written before, many early results will just be noise. They will be caused by correlated activities even though the activities are not unrelated. With an influx of new users, you can always find relationships even though they may not matter.
• When you are starting a business, you can use metrics to explore the market. Such applications include investigating an assumption, look for causalities, validate customer feedback, etc.
• In the early stages, metrics can also help with reporting if used responsibly. You can measure customer delight, the impact of new features and overall progress.
• For a new business, the first thing you should do is segment your users into cohorts (groups of people who share the same attribute). By using different segments, you may see trends or other important data that is easily missed (or offset) when you are looking at the general population.
• Klinger proposes an “AARRR” framework for looking at your early metrics. In the following, the one to reallly focus on is retention. Retention is a good indicator for how much people actually need your product. If you have retention, you can enhance monetization but if you do not have retention, it is virtually impossible to fix.
  1. Acquisition. Visits, signups, etc.
  2. Activation. Use of core features.
  3. Retention. Percent of users returning in 1, 7 and 30 days.
  4. Referral. Virality, k-score, invite and sign-up.
  5. Revenue. Self-explanatory, but includes subscriptions and in-app purchases.
• The next phase to measure is growth. Although Klinger does not say it directly, here you need to optimize LTV to exceed your cost of acquiring a new user (see my discussion of LTV). You need to ensure that LTV, which is a function of retention, monetization and virality exceeds your cost of a new user.
• Given the importance of retention, Klinger also suggests that you build a retention matrix. Along the horizontal axis, you put each of the first eight weeks (week 1, week 2, etc.) and along the vertical axis you mark each weekly cohort (see slide 84 if this does not make sense)
• One of Klinger’s most valuable points is that metrics need to hurt. If they do not hurt, they are not helping. A startup should only focus on one core metric to optimize and your dashboard should only show metrics you are embarrassed about.
• To be practical, in addition to the internal dashboard have a second one for investors and press that tells a happy story.
• Churn is a key metric to look at (when do people stop using your product). Track it at day 1, week 1, month 1 (that is the first day after installation, one week after installation, etc). Knowing when a user churns indicates why they are leaving your product.
• Klinger also make the point that I discussed in a previous post about Nate Silver, that data is very noisy. It will change a lot day to day and trying to create a cause from the noise leads to mistakes. It is much better to look at trends rather than individual data points.

Klinger’s presentation is a great primer on using metrics in a startup. Coupled with a deep understanding and application of customer lifetime value, you have the building blocks in place for success. Here is the full presentation:

Author: Lloyd Melnick

I am GM of Chumba at VGW, where I lead the Chumba Casino team. Previously, I was Director of StarsPlay, the social gaming vertical for the Stars Group. I was also Sr Dir at Zynga's social casino (including Hit It Rich! slots, Zynga Poker and our mobile games), where I led VIP CRM efforts and arranged licensing deals. I have been a central part of the senior management team (CCO, GM and CGO) at three exits (Merscom/Playdom, Playdom/Disney and Spooky Cool/Zynga) worth over $700 million. View all posts by Lloyd Melnick