While almost everyone now accepts the value of analytics and metrics-driven decision making, one area where it is often neglected is in implementing innovation. Even data driven companies are hampered in implementing innovation because their data is backward looking. In the absence of sufficient data to inform decisions about proposed innovations, managers often rely on their experience, intuition and conventional wisdom, and none of these is necessarily relevant. Although many of my readers are from the mobile game space, where much is tested, even in the game space pure innovation often is not. An article in the Harvard Business Review, “Increase your chances of success with innovation test-drives” by Stefan Thomke and Jim Manzi, does a great job of showing how to test these hypotheses.
Most companies do not conduct rigorous tests of their risky overhauls because they are reluctant to fund proper business experiments and have considerable difficulty executing them. Although the concept of experimentation is straightforward, there are many organizational, cultural and technical challenges to implementing experiments. While running an A/B test on a website is simple, many business need to deal with complex distribution systems, sales territories, bank branches, etc. Business experimentation in such environments suffers from many analytic complexities, most importantly that sample sizes are often too small to be significant (e.g., only a few stores).
Thomke and Manzi point out that to obtain actionable data on an innovation initiative and ensure that experimentation is a net positive, you should address several questions:
- Does the experiment have a clear purpose?
- What do you hope to get out of the experiment?
- What changes would be made on the basis of the results?
- Have stakeholders made a commitment to abide by the results?
- How will your organization ensure that the results are not ignored?
- Have you gotten the most value out of the experiment?
- Does the experiment have a testable prediction?
- What is the necessary sample size?
- What measures will be used to account for systemic bias?
- Does the control group match your test group?
- Have any remaining biases been eliminated through statistical analysis or other techniques?
- Do you understand what variables are impacting what metrics?
- Have you considered a targeted rollout to concentrate investment where the potential payback is the highest?
- Have you implemented only the components of the initiative with the highest return on investment?
Purpose
In determining if an experiment is needed, you first must determine what you need to learn. A weak hypothesis (e.g., “We need to offer better service”) does not present a specific independent variable to test on a specific dependent variable. A good hypothesis helps delineate those variables.
You should also consider, and test, ancillary effects of your initiative. If you are adding breakfast items to your menu, you not only should test ROI but how adding these items impacts other meals.
Buy-in
Before any experiment, stakeholders must agree how they will proceed once the results are in. They should promise to weigh the entire findings and not just pick out the ones that support what they want to do. Often the most difficult element is a commitment to abandon an initiative if the data does not support it. A process should be instituted to ensure the test results are not ignored, even if they contradict your’s or other leader’s intuition.
When constructing and implementing a process to filter experiments, you must ensure that the experiments are part of a learning agenda that supports your company’s priorities. The tests must fit and support your overall strategy.
Feasibility
The complexity of the variables and their interaction can make it extremely difficult to see cause and effect relationships. Environments are constantly changing, the potential causes of business outcomes are often uncertain and thus linkages between then are frequently complex and poorly understood. You need to consider if it is feasible to use a sample large enough to average out the effects of all variables except for those you are studying. The cost of a test involving an adequate sample size might be prohibitive or the change in operations could be too disruptive.
The key is determining the right sample size. You not only are testing for statistically significant results; managing sample size can help you decrease testing costs and increase innovation.
Reliability
You will probably have to make a trade off between reliability and time/cost. Thomke and Manzi suggest three ways to increase the reliability of your experiments:
- Randomized field trials. Take a large group of people with the same characteristics and randomly divide them into two subgroups.
- Blind tests. With a blind test, participants do not know a test is going on. You also do not tell people delivering the test which is the test group and which is the control group (for example, if Apple is testing a different practice at its stores, it does not tell anyone in the stores that they are involved in the test).
- Big data. Even outside the online world, where data is easily accessed and analyzed, you can use data to evaluate your experiment. Review the data of same store sales, results of similar situations, etc., to determine if it was the test that led to a variance in results or other factors.
Value
You may have gone through all the cost and trouble of running an experiment but then fail to use the results efficiently. You should take into account a proposed initiative’s effect on customers, markets and segments and then concentrate investment where the potential paybacks are highest. You should also look at the components of your initiative and deploy those that have the highest expected return (based on your experiment results). In addition, an analysis of data generated by experiments can help you better understand your operations and test your assumptions of what variables cause which effects. Without fully understanding causality, you may make a major mistake.
Successfully testing innovation
By paying attention to sample sizes, control groups, randomization, and other factors, you can ensure the validity of your test results. The more valid and repeatable the results, the better they will hold up in the face of internal resistance, which can be especially strong when the results challenge long-standing industry practices and conventional wisdom.
Business experimentation can lead to better ways of doing things. It can also give you the confidence to overturn wrongheaded conventional wisdom and the faulty business intuition that even seasoned executives can display. And smarter decision making ultimately leads to improved performance.
Key takeaways
- Innovation, like other important business decisions, should be pursued based on data and experimentation is the best way to obtain this data.
- To run a successful experiment, it needs to have a purpose, get buy-in internally, be feasible and reliable, and optimize value.
- The more valid and repeatable the results, the better they will hold up in the face of internal resistance and benefit your company.
The Business of Social Games and Casino 