The Business of Social Games and Casino

An often-better alternative to AB testing?

While AB testing is an integral element of mobile and social game development (as well development of most digital products), in many situations there is a better option. Several years ago, I had the opportunity to serve as an advisor to a company that had some brilliant people. Their CTO was a strong advocate of using multi-armed bandit testing as a superior alternative to AB testing. Multi-armed bandit testing is not new, there was a popular post in 2012 (http://stevehanov.ca/blog/index.php?id=132), and it is used by Google and other tech giants, but people (especially product managers) still regularly default to traditional ABn testing.

The problem with AB testing is that you leave money and performance on the table. Until the test is over, the poorer performing variant(s) will always get a significant share of your traffic. With the multi-armed bandit approach, you allocate increasingly less traffic to poorly performing variants.

What is multi-armed bandit testing

A multi-armed bandit approach allows you to dynamically allocate traffic to variations that are performing well while allocating less and less traffic to underperforming variations. Instead of two distinct periods of pure exploration and pure exploitation, bandit tests are adaptive, and simultaneously include exploration and exploitation. As Optimizely wrote recently, ” multi-armed bandit optimizations aim to maximize performance of your primary metric across all your variations. They do this by dynamically re-allocating traffic to whichever variation is currently performing best. This will help you extract as much value as possible from the leading variation during the experiment lifecycle, so you avoid the opportunity cost of showing sub-optimal experiences.”

Multi-armed bandit testing is a Bayesian approach to AB testing. As Shawn Lu writes in a post titled Beyond A/B testing, “The foundation of the multi-armed bandit experiment is Bayesian updating. Each treatment (called “arm”) has a probability of success, which is modeled as a Bernoulli process. The probability of success is unknown, and is modeled by a Beta distribution. As the experiment continues, each arm receives user traffic, and the Beta distribution is updated accordingly.”

A recap on ABn testing

To compare bandit testing with ABn testing (AB is with two variants, a test and control, n allows for additional variables), let’s quickly recap how AB testing works. Alex Atkins summarizes it succinctly, writing “in statistical terms, a/b testing consists of a short period of pure exploration, where you’re randomly assigning equal numbers of users to Version A and Version B. It then jumps into a long period of pure exploitation, where you send 100% of your users to the more successful version of your site.”

Benefits of multi-armed bandit testing

Bandit algorithms try to minimize opportunity costs and regret (the difference between your actual return and the return you would have collected had you deployed the optimal options at every opportunity). Rather than letting an AB test run until it is statistically significant, a bandit test moves subjects into the best performing group faster, allowing you to capture more gains. Matt Gershoff writes, ““Some like to call it earning while learning. You need to both learn in order to figure out what works and what doesn’t, but to earn; you take advantage of what you have learned. This is what I really like about the Bandit way of looking at the problem, it highlights that collecting data has a real cost, in terms of opportunities lost.”

A related advantage of multi-armed bandit testing is you make fewer mistakes. An A/B test will always send a significant portion of traffic to the sub-optimal group.

Also, as Shawn Lu writes, “[an] advantage of bandit experiment is that it terminates earlier than A/B test because it requires much smaller sample. In a two-armed experiment with click-through rate 4% and 5%, traditional A/B testing requires 11,165 in each treatment group at 95% significance level. With 100 users a day, the experiment will take 223 days. In the bandit experiment, however, simulation ended after 31 days, at the above termination criterion.” if the treatment group is clearly superior, we still have to spend lots of traffic on the control group, in order to obtain statistical significance.”

Finally, while not mathematically an advantage, bandit testing relieves the pressure to end a test too early. With ABn testing, frequently you will see one option perform better “directionally” and decide, or be forced to decide, to terminate the test and move everyone to the higher performing bucket before you get significant results. Unfortunately, this sometimes leads to picking an option that would be reversed once there is more data.

Why multi-armed bandit is not always the correct approach

The value of bandit testing does not mean you should abandon completely ABn testing. In Lu’s post, he writes “the convenience of smaller sample size comes at a cost of a larger false positive rate.” That is, you end up sometimes gravitating to the sub-optimal solution.

Alex Atkins also writes, “in essence, there shouldn’t be an ‘a/b testing vs. bandit testing, which is better?’ debate, because it’s comparing apples to oranges. These two methodologies serve two different needs.”
A/B testing is a better option when the company has large enough user base, when it’s important to control for type I error (false positives), and when there are few enough variants that we can test each one of them against the control group one at a time.”

The Bandit Option

While multi-armed bandit testing is not always a better option than ABn testing, you should look closely at using bandit testing when possible. It can reduce the opportunity cost of your testing and relieve pressure to terminate tests prematurely.

Key takeaways

While AB testing is the most common method of optimizing between alternatives, in many situations the multi-armed bandit approach is optimal.
A multi-armed bandit approach allows you to dynamically allocate traffic to variations that are performing well while allocating less and less traffic to underperforming variations.
Multi-armed bandit testing reduces regret (the loss pursing multiple options rather than the best option), is faster and lowers the risk of pressure to end the test prematurely.

Confirmation of Confimation Bias

Although I have written many times about different kinds of biases, the one I find most common, even in my decision making, is his confirmation bias. An article I recently came across, Confirmation Bias: Why You Make Terrible Life Decision by Nir Eyal, confirmed to me what I suspected, confirmation bias is more pervasive than most people realize. Confirmation bias is where people pick out anecdotes or facts that support their belief, while neglecting conflicting evidence.

I see if often in the games industry, you want to add a new feature (say a chat system) and you point to three products with chat systems that are highly successful. You do not look at the ten products with chat systems that have failed. Maybe you are looking to build a new product and you want to license an expensive IP. You justify it by pointing to the revenue that Kim Kardashian’s game generating while not including in your calculation the 20 branded games that failed.

This problem is not limited to the game industry. You may believe that social democracy is the best path forward for a country and you use Sweden, Norway and the Netherlands to confirm your belief, while not noticing Cuba, Greece and Spain. Or you believe free markets are the answer and confirm this by looking at Singapore, Taiwan and the US while not noticing Sweden, Norway and the Netherlands. Sometimes people knowingly pick the cases that prove their point, however, confirmation bias is when they sub-consciously accept the evidence that supports their beliefs.

Data is not the panacea

Using data rather than emotions appears as a vaccine against confirmation bias but data can contribute to the problem. Analysts suffer from the same biases as other people and will often look at the data that confirms their hypothesis.

You may have launched a new product that has a strong LTV and is growing rapidly, surpassing your existing product. One analyst who belongs to the team that launched the new product might look at the retention and monetization metrics and compare them with the original product. These KPIs are higher in the new product so the analyst recommends marketing funds shift from the older product to the new product. An analyst on the team for the existing product might identify that shows that total revenue (new plus old product) is flat since the launch of the new product, despite additional marketing spend on the new product. They might argue that the new product is simply cannibalizing the existing product. The better KPIs of the new product are a result of the best users moving to the flashy new toy, not a fundamentally better product.

Depending on the analyst’s initial partiality, they will either investigate and then present the first or second data set. Neither data is incorrect but the conclusion and actions they lead to are very different. Even though both analysts are being honest and believe they are objective, confirmation bias is driving their analysis.

Confirmation bias can impact career decisions

Confirmation bias is not only prevalent in deciding what decisions to make in business but also how to manage your career. You might feel your company is not treating you fairly. Then when two colleagues get large bonuses and you do not, it confirms that you are being treated unfairly. The data you may not be considering is that you are on a higher compensation level already or had received a bonus six months ago.

You also may be considering moving to a competitor. You have met people at a trade show from the other company and they mentioned some of the great perks. You interview and are then offered a position. Before accepting, you see some negative reviews on Glassdoor. You have already decided that you want the new job so you convince yourself the reviews have to be from a different business unit or boss. When you get to the new job, you learn the problems are real.

How do you fight confirmation bias

Given the prevalence of confirmation bias, it is important to create a strategy to combat it. A recent article, Facts Don’t Change People’s Minds, Here is What Does by Ozan Varol, provides some great suggestions:

Do not feel your beliefs create your identity. Do not get defensive when someone questions you or your project. The data is not personal and you are not a better or worse person if your hypothesis is wrong.
Develop better empathy. If someone disagrees with you, it is because they think they are correct. Understand why they are disagreeing and be open to them being potentially correct.
Get out of your echo chamber. As Varol writes, “make a point to befriend people who disagree with you. Expose yourself to environments where your opinions can be challenged, as uncomfortable and awkward as that might be.” Seek out data that disproves your position.

Additionally, I have found pre-mortems a very useful tool to combat confirmation bias. A pre-mortem is a meeting held before a major decision where all those involved in making the decision imagine themselves six or twelve months after the decision was taken, assume it turned into a debacle, and then explore why it was a disaster. This type of meeting forces you to look at contradictory facts and raise potential problems.

It is important to be cognizant of confirmation bias and seek out all the information before making important decisions. Most importantly, look at contradictory information and do not discount it because it does not support your position.

Key takeaways

Confirmation bias is where people pick out examples or facts that support their belief, while neglecting conflicting evidence.
Being data driven does not avoid confirmation bias, as people gravitate to the data that supports their beliefs.
To combat confirmation bias, do not equate your beliefs with your identity, understand why others have different positions, seek out non-confirming data and do a pre-mortem for important decisions.

When to use qualitative research

In the mobile gaming space, being data driven is largely synonymous with using quantitative data, with qualitative data (such as user research) relegated to third class status. Most game companies believe their decisions should be based on user activity in their product. They often act and allocate resources only to analyze their existing data, with design thinking and user research considered old fashioned and not as accurate. These companies, however, are missing an opportunity.

As I have said before, all data should be used to make decisions and optimize your product. Gameplay data is extremely valuable but so are qualitative data sources. Young children learn quite early that when you add a positive number to a number, the new number is larger. Thus, there is no way adding qualitative data to your quantitative data can make your analysis weaker. As long as the qualitative research does not detract from the analysis, it makes you stronger.

There is a great article, Qual vs Quant: when to listen and when to measure by Laura Klein, that explains when qualitative research is most useful. The article explains that you should constantly use both qualitative and quantitative data, but different situations require different focus.

When to focus on qual research

Qualitative research is important to understand why you are experiencing a problem or situation. Quantitative research tells you that you have a problem and what the problem is (i.e. D1 retention is down 20%).

What method in which situation

Klein highlights three different high level scenarios and then discusses whether qualitative or quantitative research would help most. In the first scenario, you are only considering changing one variable. You may be deliberating whether to change the maximum bet on slots in your casino. In this situation, quantitative analysis is optimal. With a change this small, users in a testing session or discussion will not give you any actionable information. Qualitative feedback will not provide failure to compensate for the time and money it takes to set up interviews, talk to users, and analyze the data.

The second scenario is when you are planning a multi-variable or flow change. You may be planning to implement a new feature, like a progression system, that would among other things require you adjust the user interface and journey. While you could launch it and measure performance, you would never know why it failed or succeeded. Qualitative research allows you to learn before launching what elements of the feature are compelling, what is confusing and how you can optimize the feature before you launch the feature.

The third scenario is prioritizing your roadmap. In this situation, a combination of qualitative and quantitative research is optimal. As Klein writes

The key here is that you want to look at what your users are currently doing with your product and what they aren’t doing with it, and you should do that with both qualitative and quantitative data.

Qualitative Approaches:

Watch users with your product on a regular basis. See where they struggle, where they seem disappointed, or where they complain that they can’t do what they want. Those will all give you ideas for iterating on current features or adding new ones.

Talk to people who have stopped using your product. Find out what they thought they’d be getting when they started using it and why they stopped.

Watch new users with your product and ask them what they expected from the first 15 minutes using the product. If this doesn’t match what your product actually delivers, either fix the product or fix the first time user experience so that you’re fulfilling users’ expectations.

Quantitative Approaches:

Look at the features that are currently getting the most use by the highest value customers. Try to figure out if there’s a pattern there and then test other features that fit that pattern.

Try a “fake” test by adding a button or navigation element that represents the feature you’re thinking of adding, and then measure how many people actually click on it. Instead of implementing an entire system for making friends on your site, just add a button that allows people to Add a Friend, and then let them know that the feature isn’t quite ready yet while you tally up the percentage of people who are pressing the button.

What to do

There are some situations where you should rely on the quantative research, other situations where qualitative research is best, but generally a combination of the two is the optimal way to formulate your product strategy. By using both, you are collecting all available information and using this data to drive optimal decisions.

Key takeaways

Qualitative research (survey, user panels, etc.) is often neglected by game companies who prefer to use quantitative data (KPIs) to drive decisions, but this approach neglects that more information is always better.
In situations where you are only impacting one variable, quantitative data is the answer; if you are adjusting flow or multiple variable, lean on qualitative data; and if you are prioritizing your roadmap, use a robust combination.
There are some situations where you should rely on the quants, other situations where qualitative research, but generally a combination of the two is the best way to formulate your product strategy.

It isn’t what it is

One of the most insidious phrases I have heard almost everywhere I have worked is, “it is what it is.” The phrase portends inefficiencies, politics, despair, incompetence or a lack of caring (or a combination thereof). It is critical to recognize this phrase as a problem itself and also not accept the situation it is used in reference to.

When you are likely to hear “it is what it is”

The phrase “it is what it is” generally means that the situation is how the company operates and cannot be changed. There is nothing that can be done about it. It may be that there are certain processes in place that inhibit efficiency or the ability to drive new initiatives.

The phrase will come up when someone sees an opportunity to increase profitability, maybe through a new marketing channel or by changing a software provider, only to be told the company needs to continue doing things the existing way. You may ask why is our marketing budget focuses on MySpace, and the response is we know it probably is not optimal but “it is what it is.”

You also hear the phrase when somebody is in the wrong position in the organization, cannot do their job effectively and it inhibits other people’s performance or initiatives. In American Football, a fast receiver may ask his quarterback for more long-pass attempts as he has his defender beat. The quarterback responds he would love to but the offensive coordinator does not understand the passing game. The receiver asks him to talk to the coordinator about the opportunity but the quarterback responds, “it is what it is.”

Another situation where you hear this destructive phrase is in reference to a company’s systems and software. The company may be using Windows 98. When you talk to IT about getting a newer operating system you are told they only support Windows 98. When asked why, the response is we know there are better systems out there but “it is what is.” You go back to trying to create your marketing strategy using WordPerfect.

Why “It is what it is” is so bad

The most destructive impact of “it is what it is” is not the initiative that may get stalled or the sub-optimal allocation of resources but how it destroys a company’s culture. When someone repeatedly hears “it is what it is” in response to ideas on how to improve the company, they eventually stop proposing ideas. The company then stops innovating and optimizing, which in the hyper-competitive era we live in, is the beginning of a death spiral.

”It is what is” also erodes employee morale and increases churn of your good employees. Most people work not only to make money but also because they want to achieve (most studies find that making money is not one of the key motivators for people). When the culture of it is what it is becomes prevalent, the high achievers become the most dissatisfied. There is very little worse than not pursuing an initiative they know would be good for the company not for a good reason (cash flow, strategic focus elsewhere, etc.) but because “it is what it is.” These employees are likely to find an opportunity where they can strive to be best and grow their company. This also creates a death spiral where the company is left only with people who are content to maintain the status quo.

The Kobayashi Maru

I learned a related lesson in my childhood while watching Star Trek (a show that largely shaped my adult life), a situation called the Kobayashi Maru. Not to be too geeky, but the Kobayashi Maru was a simulation given the officers at Star Fleet Academy(the training academy for people who would fly spaceships) where they had to decide between two awful scenarios, both of which resulted in a great loss of life. The point of the simulation was to show that sometimes there is not a good solution and no Star Fleet cadet ever passed.

One cadet, James Kirk (who even non-geeks probably heard of as Captain Kirk), though, did pass. What Kirk did was go in one night and reprogram the simulation, which was against Star Fleet rules. Even though it was against the rules, because he ended up succeeding where everyone else had failed, his victory was accepted.

The lesson here is that even in a no-win situation, finding a way to succeed by changing the rules. Kirk succeeded because he did not accept that “the Kobayashi Maru” is what it is. He took the initiative to make it something else and make it beatable.

Do not accept doing things sub-optimally

Start by never, ever, saying “it is what it is.” If you catch yourself about to say it, think about the situation and how you can change it. Never accept the status quo unless it really is the base path forward. If it is not, take the initiative to change the way things are done. If there is an incompetent person blocking your path, find another path. Find an ally who will work with you to drive an initiative over the blocker. If there is a better process or software solution, take it on your self to build the business case. It is easy for someone to block you by saying “it is what it is,” it is much harder from them to block you when you have a well reasoned business case in how the change will make the company an extra $500,000.

If you are in a leadership position, train your team not to accept (or say) “it is what it is.” If they are pushing an initiative forward and run into resistance, empower them to find a solution or change the rules. Only in extreme cases should they escalate the situation. It is more important that you foster a culture that your team members find a way around “it is what it is” rather than you doing it in each case (which is almost as bad). Tell your team you do not want to hear what we can’t do or are not allowed to do but want to hear how we are doing it.

When you run into “it is what it is,” rather than accepting it let everyone know the cost of not changing. If you hear it from a colleague or someone on your team, either encourage them to find a way to change things or take it upon yourself. Just as accepting “it is what it is” can put a company in a death spiral, showing it actually is not the case leads to a healthy, growing organization.

Key takeaways

”It is what it is” represents one of the most insidious phrases in the business environment, effectively telling people the status quo cannot be changed.
This phrase stops initiatives that would improve profitability, inhibits new ideas and drives away your best employees.
To combat “it is what it is,” do not accept it and build a business case to over come it.

Lifetime Value Part 28: Why you are probably under allocating resources to Live Services

Last month, I wrote a post about how many game companies do not dedicate enough resources to retention. Another area that most game companies, other than the top mobile gaming companies, are also under allocate resources to is live services. The live services team, however, largely drives success in the gaming space. A strong live services team is a key part to having an LTV that justifies marketing (and thus growth).

During the MAU conference in April, there was an interesting session on why the top social casinos were successful. The speakers were from the leading social casino publishers, Playtika, Zynga and Play Studios. All of them credited live services as the key to their success, with one attributing 80 percent of their success to live services. Also, if you look at the recent surge in Zynga’s stock price, it is largely driven by the performance of recent acquisitions. Zynga’s ability to improve the live services at these studios is the key driver of this positive performance.

What are live services

The textbook description of live services are changes that are made to a game or app that do not require a new build (development work). Live services product managers are focused on optimizing retention or monetization KPIs, rather than the PMs or designers driving new content or features.

There are many types of live services initiatives, ranging from optimizations to full programs. Some that have strong impact include:

In-game events. In many mobile games, you will see daily or weekly (sometimes hourly) special events, such as a one-day race to the top. The event may include a special leaderboard for players who get the most combinations or kill the most of a certain type of enemy in a specified period. These events serve multiple purposes. They create a sense of excitement outside the core game loop. They provide a reason for players to replay a level or a machine. They drive more engagement as players compete to get higher on a leaderboard. They provide variation, a way of introducing new content without having to build more content. Events are one of the strongest drivers of both retention and monetization KPIs in the gaming space.
Economy optimization. A key to keeping a free-to-play game successful long-term is managing the economy, an incredibly complex task. Managing the economy of a social game (or real money poker) is very similar to what Jay Powell, Chair of the Federal Reserve or Mark Carney, Governor of the Bank of England, must manage. Powell and Carney must regularly adjust interest rates, quantitative easing (asset purchases), reserve requirements, borrowing levels, etc. to manage inflation, demand, unemployment and the asset (stock) markets. If Powell or Carney gets it wrong, inflation could get out of control, unemployment could rise or asset prices (stocks and real estate) could plummet.
It is the same in games. The Live Services team must ensure that the in-game economy stays in sync and is a good experience for all players while encouraging spend (keeping up the value of assets). They need to ensure prices for goods or bets in a casino are at a price level that gives players a fair return on their investment and keeps them engaged. They need to build a balance between the main and premium currencies. They need to ensure the spin speed of a slot creates a good experience without draining a player’s wallet. They need to guarantee that a new player has a good experience but elder players are still having fun. All of these initiatives are connected and a failure in one area could create the gaming equivalent of Venezuela.
Purchase package optimization. Related to optimizing the economy, the live services tem must ensure continuously players are getting appropriate value for their purchases. If packages are priced incorrectly, a customer might not get sufficient value when they make a purchase (for example, five minutes of gameplay rather than one hour) and thus become less likely to make future purchases. Conversely, they may get so much value that they never have a need to make additional purchases.
Challenges. One of the most engaging features in games are challenges. Challenges are usually offered on a daily or weekly basis, helping direct gameplay. They are useful for keeping players engaged, encouraging them to test new content or features or play more. The live services team should create effective challenges and structure them (rewards, timing, amount of effort required, etc.) to optimize the impact on KPIs.
Sales and promotions. Just as in the retail space, sales and promotions are a valuable driver of monetization. If not structured properly by the live services team, the sales could end up cannibalizing purchases or negatively impacting the economy. When done well, they encourage higher sustained revenue.
Subscriptions As I have written recently, subscriptions are a great opportunity for game companies. Managing the pricing and options available for the subscription model sits with the live services team.

This list is a subset of some of the projects that are driven by a good live services team. Strong live services product managers will proactively identify other areas of optimization that will improve retention and monetization.

While I previously said that the live services team drives improvements that do not require development work, that is often the case in theory only. Many games are not architectured in a way that these changes can be made without development, especially older products. The need for development work should not be a line in the sand on whether these initiatives are pursued by the live services team, the benefits of launching events or optimizing the economy persist regardless of any need for development. Additionally, most successful live services teams also impact the product development efforts, for example ensuring the pricing of a new feature enhances the existing product.

Where’s the love

Just like it’s cousin retention marketing, many companies do not allocate sufficient resources (financial and people) to live services. Most of the resources end up going to product development instead. This allocation occurs because the development side is on the face of it easier to measure and sexier. A new feature under development might have a projected impact of a 2% lift on revenue after six months of development. That 2 percent uplift would be incorporated in the financial projections and when the feature launches and the revenue accelerates, everyone is taken to dinner. The PM who designed the feature can then create a Powerpoint that they use to brag to their colleagues.

Live services is not as glamorous. Economy improvements might improve revenue 0.5% every month but over the course of the month, not overnight. The live services PM probably won’t be taken out to dinner for the 0.5% increase (especially as it does not happen overnight but gradually over the month) over a given month. Over the same six months that it took to develop the larger feature, however, the economy improvements generate over 3 percent uplift (assuming the improvement is compounded monthly), a 50 percent larger increase than the new feature. Additionally, it does not require the development resources (and costs) that the new feature absorbed.

The other factor inhibiting allocation of sufficient resources to live services is the cross over with marketing. Many of the live services activities described above fall under marketing at some game companies. As marketing is often focused on acquisition, it is not prioritized on the marketing team’s agenda. Even when it is, live services need to be integrated with the core game experience to succeed. The expertise to design and optimize live services is usually more consistent with Product Managers than Marketing Managers.

Live services is critical to managing your LTV

I have written many times about lifetime value (LTV), and as I have said before it is the lifeblood of any app or game. Products are successful when their LTV is greater than the cost of acquiring a new user (CPI), only when this happens can a company afford to market. Without marketing, products eventually wither and die. Most companies, particularly in the social gaming space, are fighting a perpetual battle to find acquisition channels where LTV is great than CPI. Live services drives continuous improvement of LTV, thus allowing products that would otherwise not justify acquisition thrive. It is often the big difference between the game companies that can maintain their franchises (see Words with Friends, Slotomania, Clash of Clans, etc) to the ones that rise and then burn out.

Key takeaways

Live services is the key to the success of the largest mobile game companies (Supercell, Zynga, etc.) but it is an area often over under resourced at other companies.
Live services projects include in-game events, economy and purchase package optimization, sales, challenges and subscriptions.
Live services often get fewer resources than product development because a new feature that takes six months to develop and generates a two percent revenue uplift is sexier and easier to visualize (and put in a P&L) than improvements that add 0.5 percent a month, though the latter leads to a 50 percent bigger impact.

Building a strategy to reduce wasting time

If you read my blog regularly, you know my arch enemy is meetings. Companies lose millions of dollars on meetings in wasted salaries and the hidden cost is even higher, with people who could be growing the business instead sub-optimizing their time. In addition to meetings, the structure of many companies leads to inefficient use of their senior leaders. I recently came across a classic article published by McKinsey, Making Time Management the Organization’s Priority by Frankki Bevins and Aaron De Smet, that highlights how an organization can improve its time management.

Many people are working long days, checking email and Slack in the evening and on weekends, and still finding they cannot get their work completed. Leadership time often gets treated as though it were limitless, with all good opportunities receiving high priority regardless of the leadership capacity to drive them forward. Bevins and De Smet point out that companies should not look at this situation as a personal-productivity issue but an organizational issue that can be improved systematically.

There are multiple causes exacerbating the problem of poor time management, many of which were highlighted by Bevins and De Smet

Initiative overload, where side projects and strategic initiatives are added to a leader’s day job.
No clear direction for new leaders, who start a job without understanding what meetings to attend (and not to attend) and how to prioritize their diary
Meeting creep, as more and more people are added to regular meetings and nobody is deleted
Leaders feeling they need to be included in all transactional decisions
The multiplier effect as companies grow. As organizations grow, new verticals are formed and the leaders feel they must consult with regular 1:1s with each other. Thus, every unit added adds separate meetings for all leaders.

How to build a time management strategy

To combat inefficient use of time, Bevins and De Smet suggest several ways to build a strategy for better time management. First, there should be a leadership budget for projects. Rather than adding these projects chaotically, companies need to consider how much oversight they will require and set a time budget for approved initiatives.

Second, companies need to consider time when creating organizational change. According to the article, “companies typically look at managerial spans of control from a structural point of view: the broader they are, the fewer managers and the lower the overhead they need. Augmenting that structural frame of reference with the time required to achieve goals is critical to the long-term success of any organizational change. The hours needed to manage, lead, or supervise an employee represent a real constraint that, if unmanaged, can make structures unstable or ineffective.”

Third, companies need to ensure that their leaders measure and manage their time. By creating an atmosphere where leaders must measure how well their time allocation meets their individual strategic objectives. Whatever gets measured becomes a priority.

Fourth, companies need to review their calendar and decide what meetings support strategic goals. According to the article, “companies can make even more progress by identifying which meetings are for information only (reporting), for cross-unit collaboration (problem solving and coordination at the interfaces), for managing performance (course-correcting actions must be adopted at such meetings, or they are really just for reporting), or for making decisions (meetings where everything is approved 99 percent of the time don’t count, since they too are really for reporting). Executives at the highest-performing organizations we’ve seen typically spend at least 50 percent of their time in decision meetings and less than 10 percent in reporting or information meetings.”

Tackle the time management issue

Ironically, many companies do not tackle the waste and inefficiencies from poor time management because they cannot find the time to address it. Improving time management is not sexy like entering a new market or launching a new product but the long-term benefits can be huge.

Key takeaways

People are working long hours as well as checking email and tools like Slack off hours yet often still do not have enough time for their important projects, highlighting the poor time management in many companies.
This time management problem is created by initiative overload, no clear priorities for new leaders, meeting creep, leaders being included in meetings about ALL decisions, and organizational growth adding new layers of meetings.
To allocate time efficiently, companies should budget time for new projects, consider time demands with changing organizational structure, ensure leaders measure their time and review their corporate meeting calendar.

Why iGaming is not doomed, but doomed to be like the airline industry

Last week, a post on LinkedIn that the iGaming (real money online gambling) industry is in trouble gained traction but the article missed the reality of the iGaming space. Rather than doing poorly, the industry is becoming normal. In many ways, it is following the evolution pattern of the airline industry, which grew from a highly profitable business that succeeded largely due to regulation and barriers to entry to a very challenging business where only good companies do well.

Trouble is a relative term

Rather than being “in trouble,” the iGaming industry is suffering from unrealistic expectations. William Hill, an iGaming stalwart that was called out in the post, for the first half of 2019 had online revenue of over $425 million. While they do not report EBITDA, a similar metric would have shown an EBITDA profit of about $285 million.

The Stars Group, another industry behemoth, has seen its stock price hammered in the past year, announced EBITDA (earnings before taxes, interest and depreciation) of over $236 million and a margin of 37.1% last quarter. If you annualize that profit (EBITDA), it suggests earnings of nearly $1 billion. The performance of the other top gaming companies (Paddy Power, GVC, etc.) is in line with Stars and William Hill.

The numbers above show that iGaming companies are not hemorrhaging cash or on the verge of bankruptcy. They are still profitable large entertainment companies. They are not going away. There are very few companies in the world that would not take $1 billion in profit and a 37% margin.

So why is the iGaming business described as in trouble? Why are the share prices of most of the companies near 1-year and 3-year lows? Why are CEOs getting replaced faster than Tom Brady throwing touchdowns?

The negative sentiment is driven by high expectations, excessive debt, bad habits and a new business environment. The sentiment is also realistic, it reflects the reality that iGaming companies face, particularly the legacy operators.

The high expectations game

The great numbers put up by iGaming firms since the first companies went live created multiple problems for the industry.

Expectations beget expectations. The more the industry, and particular companies, grew, the more analysts and investors expected them to grow. The growth rates were much higher than other industries. Over time, growth is likely to revert to the average growth rate for all industries (reversion to the mean). Not only does this reversion disappoint investors but it also prompts the actors in the industry to make non-judicious decisions to chase unrealistic profit and revenue targets.
High margins drive competition. The core of capitalism is that capital flows to where it will have the biggest return. Capitalism creates an efficient allocation of resources. When there are “excessive profits” (an economic term referring to profit levels above the average for other industries), capital will flow to that industry, either financing new entrants or providing growth financing for existing companies. Capital will flow in until this expansion of supply drives down margins to “normal” levels. This phenomenon is shown by the hundreds of slots providers that are visible when walking around ICE or any gaming conference.
High expectations creates high debt. High expectations are not limited to investors. Companies themselves believe the growth will never slow. They then use this expected growth to project cash flow that justifies acquisitions (for example, GVC’s acquisition of Ladbroke Coral). When the growth fails to materialize, financing the cost of the acquisitions becomes a challenge.

When the reality does not meet expectations, particularly if the expectations do not shift, successful companies will be disappointing. Owners and investors will drive leadership changes, strategies will shift and new initiatives will begin often in an effort to actualize a future that will never exist.

It was too easy, too long

Another challenge facing the iGaming industry is that it has been too easy for too long. I had the luck of being in both the social and real money gaming spaces. One of the glaring differences is that for many years in real money gaming, particularly real money casino, it was very easy to make money. An operator could add slots to a website or launch a new app (they did not even have to build the slots as there are many providers with a huge catalog) and the more users they drove, the more money they made. The products were largely undifferentiated (in part as everyone licensed the same content) but the more that was spent on marketing, the more money companies made. Also, despite robust affiliate fees and seemingly high CPAs, it was relatively easy to grow products in a profitable way (unlike social casino where margins are wafer thin).

The ease of building a successful real money gaming company led to the high expectations detailed above. It also allowed companies to thrive without creating very compelling products. If you compare the user experiences of most iGaming apps, they are often years behind other entertainment or app offerings. Rather than optimizing the on-boarding funnel or building engagement loops to retain players, many Real Money gaming sites are little more than glorified spreadsheets. As consumers’ expectations continue to grow, this delta between expectations and what iGaming companies deliver helps drive the “troubles”.

The good times of the past have also allowed iGaming companies to stay in the past. It is probably the only industry that still talks about “mobile first.” Mobile is so integrated into virtually every other facet of our lives that even having to acknowledge that mobile is a priority highlights how far behind other industries iGaming has fallen.

What is next for iGaming

iGaming is evolving in a pattern similar to how the civil aviation industry developed. For many years, airlines and suppliers enjoyed very high profit levels as they were in a regulated industry. These companies grew into huge worldwide brands. While there were a few failures from the worst managed, it was a very stable business.

This stability led to an industry that diverged from its customers as it did not have to please them daily to survive. When the industry deregulated, these companies could not become customer friendly. They just did not understand their customers or have the right management in place. The airline industry still has one of the lowest levels of satisfaction of any consumer facing business.

They also went on a spate of mergers and acquisitions because of high expectations, creating debt service burdens that bankrupted many airlines (including huge global brands). The industry also experienced many challenger brands (Southwest, EasyJet, RyanAir, etc.) entering the market who displaced the legacy airlines.

While the iGaming industry is going in the opposite direction with regulation, it is following the path from where a few companies generated “excessive” profits to one with high levels of competition. It is also one where most of the legacy companies are not customer centric. Based on the lessons from airlines, we should not expect these legacy companies to improve their offering significantly (anyone fly BA lately), many of them will not survive including some of the largest (remember Pan Am) and industry profits will settle into much more normal numbers with some winners and some losers.

Key takeaways

There is significant negative sentiment around the iGaming industry, including depressed stock prices and frequent leadership changes. The reality is that the industry is still healthy with many companies enjoying solid revenue and strong margins, but the growth is not living up to expectations.
The industry is in this situation as the high growth and profits in the past led to unrealistic expectations that have driven more competitors and more debt and allowed companies to flourish despite products that are not centered around their customers.
Many of these legacy companies will fail to improve their offering significantly, many will not survive and industry profits will settle into normal multiples and growth rates.

Why you are probably under allocating resources to retention

One issue I see repeatedly with game and gaming companies, and many other industries, is under-indexing their investment in retention versus acquisition and reactivation. While there is no evil plot to neglect retention, it is the most challenging part of the life cycle to determine ROI on investments.

With acquisition marketing, most successful companies have visibility into the CPI (or CPA depending on how you look at your UA efforts) they are paying and the LTV for those players. By comparing CPI and LTV, there is a clear ROI. Companies can then allocate resources to acquisition marketing as long as the ROI is above their needed return on capital. Also, as you are probably spending thousands or even millions of dollars per month on acquisition, optimizing that spend justifies additional resources (analytics, tools, etc.) as you want to get the most out of your spend.

Reactivation is similarly easy to quantify. You look at the cost of getting the player back and then how much they are likely to spend once they are back. Not only is it easy to calculate the ROI on reactivation spend, you can justify the investment because the revenue will be higher than the investment. No executive can argue with spending $1 to get $2. The only caveat with reactivation spend is ensuring you are not spending to bring back customers who would return regardless.

While retention is your most important KPI, it often does not get the same level of love because the ROI is much less obvious. You are not adding a user and revenue. You are not bringing back a lost customer. Retention, however, warrants more investment than every other part of the business combined.

Retention spend avoids spending $1 for $0.40

Every dollar you spend to retain existing players impacts your recurring player base. With acquisition marketing, even the best apps and games are lucky to see a 40 percent D1 rate (40 percent of customers who download the app return the next day). Thus, for every dollar you spend on a new player, only $0.40 or less is contributing to your product’s performance the next day. Given that a normal (though still good) D7 retention rate is 10 percent (1 out of 10 newly acquired players play on the seven days after downloading your game), you effectively only have $0.10 left from that $1 of spend still working for you.

Conversely, virtually every dollar spent on existing players is contributing to your product for days or months. A contest that encourages people to play more touches all your players and could impact their behavior over their lifetime. You are not losing 60 percent or 90 percent even before you can do a thorough analysis.

Retention spend can make the cost of acquisition less painful

The first subject everyone talks about at conferences (and even importantly at the bar during conferences) is how expensive it has become to acquire customers. Yet despite the money and resources spent on acquiring the player, the same effort is not made keeping and delighting the player.

Many marketing teams feel their job is done once the player is in the game (or has made a purchase) and thus their efforts are focused on acquisition. The true value to the company, though, is turning those new customers into a loyal player. While it is not solely marketing’s responsibility, it is a critical part of the equation. If marketing acquires a player for $5, they then spend $7.50 and churn, it may look like (and is) a marketing success as they have achieved a 150% ROAS (return on ad spend). However, if they could use retention marketing to have that player make $7.50 purchases every month for a year, the impact is 12X better than the acquisition spend. Moreover, the cost of getting that player to become a repeat customer is almost certainly not 12 times the acquisition cost, but a fraction of that cost. Thus the ROI is significantly higher.

Retention is not just about the product

At this point, many of my marketing friends are probably saying (or thinking) that retention is the responsibility of the product team, they have already done their job. Product and marketing are no longer two distinct functions. Companies like Facebook and Uber have blitzscaled because their products were also their acquisition channels (hence the term growth hacking), thus these products were the marketing. Conversely, product can only touch players when they are in the product (though they can build systems to bring customers back). Marketing, however, can impact players any time. The player may not have been in the game a week but you can still impact them by email, text, Facebook, Snapchat, television, blimp, etc. Retention is largely about triggers, reminding customers to use a product again, and marketing must work with product to ensure customers stay engaged.

A bird in the hand…

Optimal investment is not only about maximizing return but also managing risk. There is much less risk dealing with a known entity (an existing player) than an unknown. If you do take more risk, you should only do it if you will have higher return. A risk-based approach to allocating your resources also suggests that retention is a neglected investment vehicle. With new user acquisition, the new player is a question mark. They may fit your existing LTV curve or they might over/under perform. There is little information to base your estimate (largely the performance of other players acquired from the same or similar channel).

With an existing player, it is easier to estimate the impact of additional retention marketing. You have data on what they play, how often they play, what incentives impact behavior, etc. Based on this data, you can estimate accurately the return one additional marketing dollar will bring. These estimates will track much closer to actual results than new acquisition, especially with new channels, thus reducing the volatility of your return on marketing spend.

Retailers get it

While game companies, and many tech companies, disproportionally emphasize acquisition, retailers have learned over hundreds of years that their marketing budget is better optimized for retention. Most of the advertising and promotions from retailers are focused on bringing back existing customers, not getting them into the store for the first time. JC Penney or Target or Curry’s are not focusing their marketing on getting new customers, they are focused on driving behavior from existing customers. Sales are designed to optimize repeat purchasers and getting existing customers to spend more, very few are built to bring in new customers.

Amazon Prime is arguably the biggest factor making Amazon one of the most valuable companies in the world. Prime, however, does not get many people to try Amazon. Instead it converts existing customers into ones who spend more and are more loyal. The dynamics of retail are not that different than gaming, it is just that they have learned that there is a higher return in devoting resources to existing customers.

What to do

The value of retention marketing does not diminish the importance of acquisition or reactivation but you should design your structure so it is not neglected.

Build your organization so that retention marketing is on an equal level. The head of retention should not report to acquisition. You should not have an EVP or SVP leading acquisition and a Manager leading retention. On the org chart, they need to be at comparable levels.
Ensure you have KPIs in place measuring retention, what gets measured gets done. Virtually any company is monitoring daily its acquisition spend, CPI (or CPA) and ROAS. You need to be both measuring and reviewing regularly the success and growth that your retention team is driving.
Rebalance your resources to ensure that you are optimizing your retention programs. Not only should you be running programs, but you need sufficient resources to build the campaigns, create the marketing collateral and analytic resources to review and optimize. Do not skimp on these resources for the quick thrill of acquisition or reactivation.
As well as physical resources, you need to deploy sufficient budget to retention for long-term success. Start with a basic smell test. If you are spending $1 million a month on acquisition and your retention budget is $25k, probably something wrong. Try to align all your marketing spend with the understanding that retention drives significant value at lower risk.

If you do not focus on retaining players, any success will be short lived. To become great, your entire company needs to focus on keeping your customers.

Key takeaways

While the primary focus of most game companies is user acquisition, retention marketing is often neglected. Retention marketing, however, is more important to a company’s prolonged success.
For every dollar spent on acquisition marketing, at least $0.60 is lost the next day only 40 percent (at best) of players come back. Conversely, all dollars spent on retention marketing impact customers over their lifetime in your game.
You should ensure your structure is built on optimizing retention as well as acquisition and allocate resources (both people and money) to reflect the immense opportunity with retention marketing.

December 2019
S	M	T	W	T	F	S
« Nov
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30	31

What is multi-armed bandit testing

A recap on ABn testing

Benefits of multi-armed bandit testing

Why multi-armed bandit is not always the correct approach

The Bandit Option

Key takeaways

Share this:

Like this:

Share this:

Like this:

Data is not the panacea

Confirmation bias can impact career decisions

How do you fight confirmation bias

Key takeaways

Share this:

Like this:

When to focus on qual research

What method in which situation

What to do

Key takeaways

Share this:

Like this:

When you are likely to hear “it is what it is”

Why “It is what it is” is so bad

The Kobayashi Maru

Do not accept doing things sub-optimally

Key takeaways

Share this:

Like this:

What are live services

Where’s the love

Live services is critical to managing your LTV

Key takeaways

Share this:

Like this:

How to build a time management strategy

Tackle the time management issue

Key takeaways

Share this:

Like this:

Trouble is a relative term

The high expectations game

It was too easy, too long

What is next for iGaming

Key takeaways

Share this:

Like this:

Share this:

Like this:

Retention spend avoids spending $1 for $0.40

Retention spend can make the cost of acquisition less painful

Retention is not just about the product

A bird in the hand…

Retailers get it

What to do

Key takeaways

Share this:

Like this: