Cracking the code of seasonal A/B testing

App market is not a stationary environment where apps sit and wait to be downloaded. Everything changes daily, even momentarily in this competitive market and this influences both the apps and the users. Staying on top of how outside influences and ‘high-seasonality’ can affect your ASO performance needs constant attention.

Users change their behaviours, their decision-making, interests, intentions and opinions constantly change. Additionally, apps themselves are not one-dimensional; many have multiple features and benefits that have a relevance to different audiences.

Every app vertical will have its unique ‘high-season’; dating and fitness apps might go all-in at new year; betting apps capitalise on new football seasons or horse racing events; eComm waits all year for Black Friday. All seasons may form the basis for a viable A/B test opportunity.

However, the most important thing is to know your audience and their interests and to remember that regionality must also be considered: interest seasons may differ. While Christmas is an important event in most EU countries, it might be less so in countries with other religious majorities; the ‘back to school’ season differs by country, as do Mothers’ and Fathers’ Days; there is a global array of national holidays - so do your research and plan accordingly.

Here are the typical seasons that could offer opportunities for many apps to A/B test; remember, events and seasons change globally; some events are celebrated differently, or on a different day, depending on country or region.

• Lunar New Year
• Spring, summer, autumn, winter seasons
• Sporting events and season-openers
• Pride
• Black Friday / Cyber Monday
• Mothers' and Fathers' days
• Thanksgiving, Christmas
• New year
• Back-to-school moments
• Halloween
• Saints days, such as St Patrick's and St Valentine's days

All these variables might seem like a massive headache, but actually, if done right, they present an opportunity for apps to create an uptick in UA by continually evolving the product offering and optimising their app market listing to stay relevant to users and the moment, and to stay ahead of the competition. But before you head for the drawing board, have a good think about how you'll meaningfully measure which variables are moving the dials for your app.

A/B testing is the solution and in the context of a competitive app vertical, A/B testing offers an invaluable opportunity to understand how users react to your product page changes; it presents a strategic way to focus on the seasonal (or otherwise) variables that keep your app converting and engaging.

Imagine being able to make data-driven decisions that enhance conversion rate, user engagement, retention, and revenue generation. This is where A/B testing comes into play, allowing game developers to fine-tune their apps and tap into the potential of different seasonal variations to maximise UA performance.

Below are some product page elements that you can change (and test) on a seasonal basis:

You can utilise the opportunities below to improve your visibility in different seasons

• Custom Store Listings (impacts CPI, paid CR)
• Custom Product Pages (impacts CPI, paid CR)
• Promo Text (impacts page views to install CR)
• In-App Events

Here, an example of generic screenshots for a solitaire app, alongside screen shots for the same app, tailored to the festive season, when people might have more free time to play the game.

Here, an example of how a generic graphic might be subtly tailored towards the festive season, to be more relevant to people searching for a solitaire app, at that time of year.

ASO is not only about making optimisations and satisfying the app market algorithms; it’s about joining together and capitalising on the investments you have made across the other app market disciplines and other media (paid UA, above-the-line media, PR and product development); ASO stands at the heart of converting search and it connects each discipline to the other. So while making your seasonal A/B test decisions, and creating your IAE and custom product page place, you’ll need to consider and plan for multiple factors.

Make sure that you check-in with your product and development teams before you plan your tests. Any changes that they are making to the product may impact your test. You need to ensure that your storytelling aligns with the product offering and that your marketing tone of voice is the same as the user will experience in-app. Stay up to date with your product roadmap, know when new in-app UX/UI changes are happening or new features are coming (or when bugs are fixed).

Your product page and other promotional assets like IAEs should match with your marketing effort. Creative assets and content that your marketing team will utilise in their wider campaigns should match with your product page and tell the same, cohesive story.

Use ASO tools at your disposal to deep-dive into your competitors’ app store listing, performant keywords and creative strategies around events and seasons. Understand and utilise that data when planning your own activities.

Build a simple testing framework to facilitate cross-team collaboration and to implement, track and assess your A/B test assets; such as roadmap with seasonal event names, target countries, hypothesis and result expectations.

Keep in mind that you have a limited time to test before an event comes and goes, and users’ search and conversion behaviours will change only for a limited time. So you might want to consider some of the following constraints, as they may impact your ability to collect enough data to deliver meaningful insights:

• Keeping your test group traffic percentage low will cause your test to take more time to collect enough data to reach statistical significance; with a short event, this could be an issue.
• Testing more than 1 asset at once will require more installs to gain enough data to assess all your assets. Accumulating these installs will take time and increase your testing duration.
• Carefully pick your start date: if you start way before your key season your assets won't be user-relevant and may not convert well. You also run the risk of overlapping with other seasons.
• Carefully pick your end date: you need to monitor your data and allow your test to run for the optimal time period, before relevancy drops away.
• Know when to stop: you can’t always win A/B tests; in most cases your test asset might convert worse than the default one. Agreeing when to stop your test if things don’t go as planned, is key.

Depending on our app's daily install potential, you can even quickly adjust the test assets and run another test until the seasonal event comes up, if your first seasonal test asset fails.

We recommend keeping your tests at least for 14 days to be able to understand how the test asset performs on different days of the week. However, if the difference between your test asset and default asset is consistently growing in the default asset’s favour, this can be a sign for you to stop your test.

The app market is constantly changing, and apps must adapt to these changes. User behaviours, interests, and opinions also evolve, influenced by factors like seasons, sports events, holidays, and special occasions. The smartest app marketer uses data to make the decisions that affect budget, UA and monetisation. Gathering that data is a constant challenge; failures will happen, but an effectively planned A/B testing can provide learning either way, and ultimately a roadmap to improved performance. The only mistake you can make is not to test at all.

If you feel your app is in need of an ASO audit, or could scale with the right seasonal strategy, contact us. We've got a great team of experts who can do the heavy lifting and the big thinking for you!