I recently read a paper, “The Golden Rule of Forecasting: Be Conservative” by Armstrong, Green and Graefe, that showed empirically the most important principles in making forecasts and predictions. Given the value of accurate forecasting (e.g., for building your business, making investments, choosing between product strategies), by understanding the golden rule you will help optimize your decision making.
What is particularly compelling about this paper is that it is based on extensive research and empirical studies. So while many forecasting and decision-making guidelines are based on hypothesis or observation, The Golden Rule of Forecasting is based on data.
At the heart of the golden rule of forecasting is that you should be conservative; forecasters must seek all knowledge relevant to the problem and use methods that have been validated for the situation.
With “be conservative” as the overarching golden rule, the authors performed extensive research to develop a list of guidelines that help lead to good forecasts and identified practices that generate poor decisions. Below is a summary of those guidelines that are most relevant to game companies, tech startups and hopefully all readers of this blog. The more of these guidelines you follow, the more accurate your forecasts.
- Use all reliable, relevant and important information, and no more. One way to achieve this guideline is by asking experts to list the relevant variables, the direction and strength of their effects and recommendations on which data are relevant to what you are trying to predict. You can also use literature to find what variables are critical.
- Use recent data.Conservative forecasting requires knowing the current situation, so you should seek out the most current data. If you are looking at green lighting a game project, it is much more important what happened last quarter than two years ago.
- Decompose the situation. Rather than just looking at what you are trying to forecast holistically, and thus trying to find prior data of a similar situation to guide you, you will generate more accurate forecasts by decomposing the problem you are trying to forecast. Decomposition is conservative because the errors from forecasts of the parts are likely to differ in direction and thus offset each other in the aggregate. It also allows you to use different forecasting techniques for different parts of the problem.
- Avoid bias. Forecasters sometimes depart from prior knowledge due to biases they may be unaware of, such as optimism, or using the most familiar methods and accessible data. Financial and other incentives, deference to authority, and confusing forecasting with planning or motivation can cause forecasters to ignore prior knowledge or choose invalidated methods. I have often seen bias come into sales forecasts, which are crucial for staffing and cash flow planning, when people try to create revenue forecasts to please superiors. Some ways to avoid bias is to conceal the purpose of the forecast from the person making the forecast. Another way is to make forecasts on multiple different hypotheses (such as forecast four different game types). One other technique is to ask the person making the forecast to sign an ethics statement. This option may sound silly in the business environment but consider asking your analyst to sign their next forecast on a new feature.
- Provide full disclosure to encourage independent audits. Replications are fundamental to progress and building on results. Additionally, the threat of an audit in itself may encourage the forecaster to follow evidence-based procedures.
- Avoid unaided judgment. Structured judgments follow validated procedures to make effective use of available knowledge. The authors point out that unaided judgments, however, are not conservative because they are a product of faulty memories, inadequate mental models, and unreliable mental processing. Unaided judges tend to see patterns in the past and predict their persistence without data to back it up.
- Use structured analogies. A situation of interest, or target situation, is likely to turn out like analogous situations. Using evidence on behavior from analogous situations is conservative because it increases the knowledge applied to the problem.
- Combine forecasts. Combining forecasts, if you keep them independent of each other, increases the amount of information considered and reduces the effect of biases.
- Use the longest time-series of valid and relevant data. The person making a prediction can influence the forecast by choosing the starting point for the data they are looking at. This choice can reinforce the analyst’s prior beliefs. To eliminate this potential source of inaccuracy, ask the person making the decision to first look at the effects of causal forces on the trend of the series to be forecast. When forecasting a time-series that is the product of opposing causal forces, such as growth and churn, decompose the series into the components affected by those forces and extrapolate each component separately.
- Be conservative when forecasting trends. Conservatism calls for a reduction in the magnitude of the trend, also referred to as “damping.” This keeps forecasts closer to the estimate of the current situation.
- Estimate seasonal factors conservatively. For situations undoubtedly affected by causal factors, such as gift cards, seasonal factors often improve accuracy. When the situation is uncertain, damp the estimated seasonal forecast. If there are only a few years of data or the causal relationship is very uncertain (e.g., the Super Bowl winner’s impact on the stock market), further dampen the projected impact of seasonal factors.
- Combine estimates from different methods or alternative data. When it is possible, use (or have your analysts use) different methods to make a forecast. Also, if different sets of data are available, use them separately to come up with different forecasts and then integrate them.
- Estimate variable weights conservatively. Damp estimates of each variable’s coefficient (weight) towards zero, or “shrinkage.”Shrinkage reduces the amount of change that a model will predict in response to causal variables, and is thus conservative when predicting change. Thus, if you see another product sees a large change in retention because it has a unique feature, when estimating how that feature will impact your product you need to be conservative.
- Use all important variables. When estimating relationships using non-experimental data, regression models typically only use a maximum of three variables but there may be many important variables in the data you are using to make a forecast. Instead of a simple regression, you may want to use an index model where you list all important variables, identify their directional effects and weigh them by importance.
- Avoid unstructured judgmental adjustments to forecasts. Making subjective adjustments to a forecast can introduce biases and random errors. Managers and analysts are often tempted to make unstructured adjustments to forecasts from quantitative methods.
If you truly want to create optimal forecasts, and base your decisions and planning on these forecasts then follow the Golden Rule of Forecasting: Be conservative by adhering to cumulative knowledge and focusing on the guidelines above.
The Business of Social Games and Casino 