A/B testing in mobile apps: what the results can and can't tell you

Every few weeks someone sends me a screenshot of a test result with a green arrow on it and asks whether they should ship the winner. Sometimes the answer is yes. More often it is "not yet", and occasionally it is "this test could never have told you that". The tools for testing on the App Store, on Google Play and inside apps are good enough now that running an experiment is easy. Reading one is the hard part.

These are the questions I get asked most, answered as plainly as I can. Store rules are as stated on Apple's and Google's own help pages at the time of writing, and they do change, so check before you plan a test around them.

What can I actually test on the stores?

On the App Store, the tool is called product page optimization. According to Apple's product page optimization guide, you can test up to three treatments against your original page, using alternate app icons, screenshots and app preview videos. Text is not part of it: no name, subtitle or description tests. Treatments are shown to a random share of people on iOS and iPadOS 15 or later, and each person keeps seeing the same version for the whole test.

On Google Play, the tool is store listing experiments in Play Console. Google's help page on store listing A/B tests describes two types: a default graphics experiment (icon, feature graphic, screenshots, promo video) and localized experiments, which can also test the short and full description in up to five languages. You can add up to three variants next to your current listing.

Two Apple details catch people out. A test cannot be changed once it has started, and if a treatment uses an alternate icon, that icon has to be included in the binary of your published app. Tests without alternate icons can go to review independently of a new app version.

How is that different from testing inside the app?

Store tests measure one thing: whether a visitor installs. In-app experiments measure what happens afterwards, such as whether people finish onboarding, start a trial or come back on day seven. You run them with a remote configuration or feature flag system; Firebase A/B Testing on top of Remote Config is a common starting point, and plenty of teams build their own.

The mechanics are more in your hands, and so are the mistakes. Assign variants per user, not per session, and keep the assignment stable across app launches and devices if you can. Decide on one primary metric before you start and a few guardrails you refuse to make worse, such as crash rate, refunds and support tickets. Then check that your split is the split you asked for. If you configured 50/50 and see a clearly lopsided count on a large sample, something in the assignment or logging is broken, and no result from that test is worth reading.

Onboarding is usually where in-app testing pays first, because every user passes through it; our piece on onboarding flows people actually finish has a list of steps worth measuring. Paywalls come second, with more care, since price and trial changes interact with everything in how an app makes money. Google Play also offers price experiments for in-app products inside Play Console, which saves building that part yourself on Android.

How much traffic do I need?

More than most people hope. The size of the difference you want to detect matters far more than anything else, and the relationship is brutal: to detect half the effect, you need roughly four times the traffic. A back-of-envelope rule from statistics textbooks makes this concrete:

n_per_group = 16 * p * (1 - p) / d^2
p = baseline conversion rate, d = absolute difference to detect

This rule assumes 95 percent confidence and 80 percent power, and the stores use their own methods, so treat it as a sanity check rather than a forecast. Here is an illustration with made-up but ordinary numbers. Say 4 percent of product page impressions turn into downloads, the page gets 2,000 impressions a day, and you split traffic 50/50 between the original and one treatment.

• To detect a lift from 4.0 to 4.8 percent (20 percent relative), you need about 9,600 impressions per group, so roughly ten days.
• To detect 4.0 to 4.4 percent (10 percent relative), you need about 38,400 per group, so roughly five to six weeks.
• To detect 4.0 to 4.2 percent (5 percent relative), you need about 153,600 per group, which is longer than Apple's 90-day maximum.

Both stores will estimate this for you. Apple's optional estimate shows the impressions and time needed to reach at least 90 percent confidence for the improvement you pick, and Play Console has a calculator based on your minimum detectable effect and confidence level. Use them before you design the treatments, not after. If the estimate is longer than you can wait, test bigger differences or fewer variants, because every extra variant splits the same traffic further.

How long should a test run?

At least as long as the sample size requires, and always in whole weeks. Store traffic on a Tuesday is not the same as traffic on a Sunday, and a test that ran Thursday to Monday has seen two weekend days out of five. I set a minimum of two full weeks even when the calculator says less.

Watch for anything that changes the traffic mix while the test runs: a feature on the store, a paid campaign starting or stopping, a holiday, a press mention. A treatment that wins during a burst of paid traffic has won with that audience, which may not be the audience you get the rest of the year.

Can I look at the results while the test is running?

You can look. The danger is acting on what you see. In a classic fixed-length test, checking every day and stopping the first time a variant looks significant makes false winners much more likely, because random swings cross the line now and then and you stop at exactly those moments.

The two stores handle this differently. Google says its store listing experiments use confidence intervals designed to allow continual monitoring, and results stay at "More data needed" until there is enough to say something. Apple recommends waiting to apply a treatment or stop a test until at least one treatment has been declared better or worse than the baseline with at least 90 percent confidence. For in-app tests, the rule depends on your tool's statistics, so write down the stopping rule before the test begins and stick to it.

Also keep the confidence level in perspective. Google's own help page notes that at 90 percent confidence, one in ten experiments may report a false positive. Run ten tests and expect a fluke among your winners. For any change you would hate to get wrong, run it again.

What is a novelty effect, and does it matter here?

A novelty effect is a change that performs well because it is new, then fades as people get used to it. Its mirror image is a change that performs badly at first because regular users dislike any change at all. Both are mostly a problem for in-app experiments on existing users, who know what the old version looked like.

Store tests are less exposed, because most visitors to a product page have never seen it before. Not entirely, though. Returning visitors exist, and on the App Store an alternate icon from a treatment also appears on the device of anyone who downloads during the test. For in-app tests, compare new and existing users separately and check whether the effect shrinks week by week. If it does, wait until it settles before you call it.

What can a winning result not tell me?

Why it won, for a start. A test tells you that one version converted better, for the audience in the test, during the test. It says nothing about which element did it if you changed several at once, which is why Google recommends testing one attribute at a time and Apple asks you to consider how many elements you change in each treatment.

It also can't tell you whether those extra installs are good installs. A screenshot set that promises more than the app delivers can win a store test and lose you money a month later. Google's retained-installer metric covers the first day. Beyond that, you need your own cohort data, and if a store winner shows weaker retention or trial conversion, I trust the downstream numbers. Hypotheses fail all the time, too. One developer story Apple publishes on its optimization page describes a test where adding a preview video as the first asset lost to the original page, against the team's expectation.

What should I test first?

Start where the traffic sees the most and the differences are large. On both stores that means the first screenshots and the icon, since those are visible in search results before anyone opens the page. Our guide to screenshots that explain the app in five seconds is a good source of treatments worth testing: a different first message, a different core screen, a different order. On Google Play, the short description is the cheapest text test, run as a localized experiment.

Leave subtle color tweaks and button shades for apps with very large traffic. Write the hypothesis in one sentence before building the assets, with the metric and the change you expect, and keep a log of every test with its dates, traffic share and result, including the draws. In a year, that log will tell you more about your users than any single green arrow.