There’s a great blog post on GamesBrief on how to get your A/B testing efforts going. Given the importance of A/B testing to optimizing both your game’s performance and user acquisition, this is a must-read article if you are not already A/B testing.
To summarize the post (read the full post for a much deeper explanation of each point), the key point is that there are six steps to start successful A/B testing:
- Start simple. It is more important that you are testing than that you are testing perfectly and everything. Start with a manageable or a few manageable A/B tests. The important thing is to make sure you have enough data to get statistically significant results.
- Avoid latency. Delay is your enemy, if it takes too much time to set up a test and implement its results, it becomes too costly (in terms of resources) or risky to test. Instead, structure your game so that engineering does not have to implement each test.
- Implement a ‘kill switch’. Things go wrong, even though you may think they will not for you. By adding a kill switch, you no longer have to worry about failing and instead can focus on your testing.
- Isolate your variables. For the results to be useful, you should only test one variable with each test. If you change multiple attributes (for example the price and color), you will not know how each impacts your lifetime value (and they could even have opposite effects).
- Check longitudinal impacts. You must run your tests over a period of time rather than act on the numbers immediately. Over time, the effects may change … or even reverse. What generates no additional monetization immediately may significantly help in the future or could destroy retention in three weeks.
- Treat new and existing users differently. Different cohorts of users will react differently to changes in the game. As I discussed in a previous post, it is important to analyze lifetime value by cohort and for the same reason you need to judge the results of the A/B test by cohort and assume if the change is made throughout the game it will most likely create similar behavior as your most recent cohort.
The Business of Social Games and Casino 