THE ASO PROFESSIONAL FIELD GUIDE / Write ASO test hypotheses
A/B TEST. NOT A/B GUESS.
“Try a different screenshot” is an instruction. A hypothesis explains who the change is for, why it might matter and what evidence would support the decision.
An ASO test hypothesis connects a specific audience, a proposed change and a plausible mechanism to an observable outcome. Before launch, add the actual platform setup, primary measure, review conditions and the decisions each possible result would support.
Your deliverable: Write a test plan with an audience, mechanism, primary measure and decision rule.
A browser-local writing aid. It does not calculate sample size, validate a metric or establish statistical significance. Your text is not uploaded.
01
State the uncertainty you want to resolve.
Begin with an observation: visitors may not understand how quickly they can start, the opening promise may be too broad, or an important use case may be hidden. Separate that observation from your interpretation. You may know the message is absent without knowing whether its absence is hurting conversion.
A useful test addresses the uncertain part. It does not exist simply to demonstrate that the team is running experiments.
02
Name the audience, change and mechanism.
For a fictional routine app, the hypothesis might be: for new visitors seeking a simple morning routine, showing a completed three-step routine first will make the value clearer than the current settings screen. The proposed mechanism is comprehension of the outcome.
That statement identifies who the change serves and why it might influence behaviour. It still needs a suitable metric, setup and decision rule before it becomes an experiment plan.
03
Choose a metric that matches the question.
Use the exact metric definition from the report or experiment tool. Document the population and any exclusions. Do not switch to a different favourable metric after seeing the result without acknowledging that the question changed.
Downstream quality can matter too, but attribution and data availability may limit what can be connected to the test. The conversion guide and calculator explain why definitions and percentage points matter.
04
Confirm that the store supports the proposed test.
Apple’s product page optimization supports experiments with icons, screenshots and app previews. Google Play provides default graphics and localized experiments, with text available in the supported localized setup. A metadata idea should not automatically be assumed testable through the same interface as screenshots.
Agree the observation conditions, primary measure and practical threshold before launch. Traffic and baseline behaviour affect how informative the experiment can be. Repeatedly checking and stopping as soon as a favourable number appears can encourage misleading conclusions.
A useful outcome can be support for rollout, evidence against the idea, or insufficient evidence. None should be rewritten into a guaranteed success story.
06
Keep the next decision visible.
Use the builder to draft a hypothesis, then add the platform setup, owners and review plan to your project record. The generated sentence is a writing aid, not a statistical design or sample-size calculation.
If you cannot explain the mechanism, return to the screenshot argument or the strategy. A better question is often more valuable than another visual variant.
WORKED EXAMPLE / FICTIONAL APP
A routine app tests an outcome, not a favourite colour
MorningKit is a fictional routine app. Its first screenshot shows a configuration menu. The team proposes a completed three-step routine because visitors may struggle to imagine the outcome. This is a testable message hypothesis; “make it more modern” would not identify the mechanism clearly enough.
How to reason through it
Name the audience and the uncertainty. Specify what changes, what stays stable, and why the change might affect behaviour. Then define the exact platform measure and the practical decision it will inform. Record constraints such as traffic, localized assets and other releases before launching. A test cannot rescue a question that changes after every result update.
The next practical move
Prepare one coherent alternative sequence and verify product accuracy before exposure. Use the platform’s available planning and result tools for the actual setup. Agree how the team will handle evidence for rollout, evidence against the idea, and an inconclusive result. Keep the result attached to this audience and treatment; do not turn it into a universal rule about all routine apps.
Swipe horizontally to compare the decision map →
A routine app tests an outcome, not a favourite colour: illustrative decision map
Decision area
What to inspect
What it means for the work
Hypothesis
Completed routine clarifies the outcome
Comprehension is the proposed mechanism
Primary measure
Exact selected platform event and population
Keep the definition stable
Decision
Roll out, retain control, or investigate further
Use the planned interpretation conditions
The trap to avoid
Do not combine unrelated changes and then attribute the result to your favourite one. A combined treatment can answer whether the package worked, but it cannot isolate the contribution of the headline, background and sequence individually.
When the situation gets more complicated
Distinguish operational deadlines from evidence thresholds. If the campaign must launch before a test can inform a decision, state that limitation and choose an operational course deliberately. Do not lower the evidence standard after seeing an attractive estimate. Keep a learning log of rejected and inconclusive hypotheses so the team does not repeatedly pay to ask the same question.
FROM METHOD TO DELIVERABLE
A test brief and decision tree for MorningKit
This fictional plan compares an outcome-led opening with a settings-led opening. It is an editorial planning example, not a sample-size calculation or a claim that the treatment will win. Resolve every setup dependency before exposing the variant.
Swipe horizontally to read the complete example →
Fictional experiment plan. The table supplies decisions, not statistical thresholds.
Plan element
Completed example
Launch or review condition
Question
Does showing a completed routine clarify the first-use outcome?
Check that the screen reflects the available product
Audience
Visitors viewing the selected English listing
Confirm the actual eligible population; do not imply targeting the tool cannot provide
Treatment
Completed routine replaces the opening settings image
Keep other planned assets stable; archive both versions
Primary measure
The selected platform-defined listing action measure
Record its exact event, denominator and population before launch
Favourable evidence
Treatment satisfies the agreed interpretation conditions
Check practical relevance and integrity before rollout
Unfavourable or inconclusive evidence
The idea lacks support or uncertainty remains
Retain the control unless a separate operational reason warrants change
Turn a vague idea into a falsifiable question
“Make the screenshots better” cannot explain what the result would teach you. “Show a completed routine first because visitors may not recognise the outcome” names a mechanism. A favourable behavioural result can support using the treatment, but it does not by itself prove that comprehension was the psychological cause. Use suitable qualitative evidence if that mechanism matters to the next decision.
Agree review conditions without inventing a universal duration
Use the actual platform planning and result interpretation guidance. Record the selected configuration, practical effect worth acting on, known audience changes and review responsibility. A weekly meeting can be a status check without becoming a new stopping rule. Do not keep changing the question or preferred measure after seeing the numbers.
Preserve an inconclusive result as useful information
An inconclusive comparison does not establish equality. Record what was tested and why the evidence could not settle the decision. Decide whether more suitable evidence, a stronger concept or a different business priority is warranted. If a deadline forces a choice, label it an operational choice and retain the uncertainty in the experiment log.
A treatment looks favourable early, then its reported range includes no improvement. Can the team publish the early uplift as a win?
Reveal the reasoning
No. Use the agreed evaluation plan and the complete evidence. An early favourable estimate is not a durable conclusion. Record the uncertainty and choose whether more appropriate evidence or a revised hypothesis is warranted.
Your working brief
Use your own app for this exercise. Fill in the six fields below and attach the evidence behind each answer. If you cannot complete a field, write the missing question and the person who can resolve it. Do not replace unknowns with optimistic assumptions.
Audience and listing context
Control and proposed treatment
Mechanism and supporting observation
Primary metric and exact event definition
Setup, competing changes and review rule
Result, uncertainty and next decision
For each field, distinguish an observation from an interpretation. Add the date and source where available, then identify the decision that follows. A colleague should be able to trace your recommendation back to the evidence without needing you to explain the entire project again.
