How to implement A/B testing

There’s a great blog post on GamesBrief on how to get your A/B testing efforts going. Given the importance of A/B testing to optimizing both your game’s performance and user acquisition, this is a must-read article if you are not already A/B testing.

A/B testing image from

To summarize the post (read the full post for a much deeper explanation of each point), the key point is that there are six steps to start successful A/B testing:

  1. Start simple. It is more important that you are testing than that you are testing perfectly and everything. Start with a manageable or a few manageable A/B tests. The important thing is to make sure you have enough data to get statistically significant results.
  2. Avoid latency. Delay is your enemy, if it takes too much time to set up a test and implement its results, it becomes too costly (in terms of resources) or risky to test. Instead, structure your game so that engineering does not have to implement each test.
  3. Implement a ‘kill switch’. Things go wrong, even though you may think they will not for you. By adding a kill switch, you no longer have to worry about failing and instead can focus on your testing.
  4. Isolate your variables. For the results to be useful, you should only test one variable with each test. If you change multiple attributes (for example the price and color), you will not know how each impacts your lifetime value (and they could even have opposite effects).
  5. Check longitudinal impacts. You must run your tests over a period of time rather than act on the numbers immediately. Over time, the effects may change … or even reverse. What generates no additional monetization immediately may significantly help in the future or could destroy retention in three weeks.
  6. Treat new and existing users differently. Different cohorts of users will react differently to changes in the game. As I discussed in a previous post, it is important to analyze lifetime value by cohort and for the same reason you need to judge the results of the A/B test by cohort and assume if the change is made throughout the game it will most likely create similar behavior as your most recent cohort.

Author: Lloyd Melnick

I am GM of Chumba at VGW, where I lead the Chumba Casino team. Previously, I was Director of StarsPlay, the social gaming vertical for the Stars Group. I was also Sr Dir at Zynga's social casino (including Hit It Rich! slots, Zynga Poker and our mobile games), where I led VIP CRM efforts and arranged licensing deals. I have been a central part of the senior management team (CCO, GM and CGO) at three exits (Merscom/Playdom, Playdom/Disney and Spooky Cool/Zynga) worth over $700 million.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: