# A routine app tests an outcome, not a favourite colour

Companion to /learn/conversion/ab-test-hypotheses/

## Outcome
Write a test plan with an audience, mechanism, primary measure and decision rule.

Fictional example, not a client result.

## Scenario
MorningKit is a fictional routine app. Its first screenshot shows a configuration menu. The team proposes a completed three-step routine because visitors may struggle to imagine the outcome. This is a testable message hypothesis; “make it more modern” would not identify the mechanism clearly enough.

## Reasoning
Name the audience and the uncertainty. Specify what changes, what stays stable, and why the change might affect behaviour. Then define the exact platform measure and the practical decision it will inform. Record constraints such as traffic, localized assets and other releases before launching. A test cannot rescue a question that changes after every result update.

## Action
Prepare one coherent alternative sequence and verify product accuracy before exposure. Use the platform’s available planning and result tools for the actual setup. Agree how the team will handle evidence for rollout, evidence against the idea, and an inconclusive result. Keep the result attached to this audience and treatment; do not turn it into a universal rule about all routine apps.

## Trap to avoid
Do not combine unrelated changes and then attribute the result to your favourite one. A combined treatment can answer whether the package worked, but it cannot isolate the contribution of the headline, background and sequence individually.

## Advanced considerations
Distinguish operational deadlines from evidence thresholds. If the campaign must launch before a test can inform a decision, state that limitation and choose an operational course deliberately. Do not lower the evidence standard after seeing an attractive estimate. Keep a learning log of rejected and inconclusive hypotheses so the team does not repeatedly pay to ask the same question.

## Exercise
A treatment looks favourable early, then its reported range includes no improvement. Can the team publish the early uplift as a win?

## Suggested answer
No. Use the agreed evaluation plan and the complete evidence. An early favourable estimate is not a durable conclusion. Record the uncertainty and choose whether more appropriate evidence or a revised hypothesis is warranted.

## Your worksheet

### Audience and listing context
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

### Control and proposed treatment
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

### Mechanism and supporting observation
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

### Primary metric and exact event definition
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

### Setup, competing changes and review rule
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

### Result, uncertainty and next decision
Answer:
Evidence:
Uncertainty:
Owner:
Next action:

## A test brief and decision tree for MorningKit
This fictional plan compares an outcome-led opening with a settings-led opening. It is an editorial planning example, not a sample-size calculation or a claim that the treatment will win. Resolve every setup dependency before exposing the variant.

Fictional experiment plan. The table supplies decisions, not statistical thresholds.
Plan element | Completed example | Launch or review condition
Question | Does showing a completed routine clarify the first-use outcome? | Check that the screen reflects the available product
Audience | Visitors viewing the selected English listing | Confirm the actual eligible population; do not imply targeting the tool cannot provide
Treatment | Completed routine replaces the opening settings image | Keep other planned assets stable; archive both versions
Primary measure | The selected platform-defined listing action measure | Record its exact event, denominator and population before launch
Favourable evidence | Treatment satisfies the agreed interpretation conditions | Check practical relevance and integrity before rollout
Unfavourable or inconclusive evidence | The idea lacks support or uncertainty remains | Retain the control unless a separate operational reason warrants change

### Turn a vague idea into a falsifiable question
“Make the screenshots better” cannot explain what the result would teach you. “Show a completed routine first because visitors may not recognise the outcome” names a mechanism. A favourable behavioural result can support using the treatment, but it does not by itself prove that comprehension was the psychological cause. Use suitable qualitative evidence if that mechanism matters to the next decision.

### Agree review conditions without inventing a universal duration
Use the actual platform planning and result interpretation guidance. Record the selected configuration, practical effect worth acting on, known audience changes and review responsibility. A weekly meeting can be a status check without becoming a new stopping rule. Do not keep changing the question or preferred measure after seeing the numbers.

### Preserve an inconclusive result as useful information
An inconclusive comparison does not establish equality. Record what was tested and why the evidence could not settle the decision. Decide whether more suitable evidence, a stronger concept or a different business priority is warranted. If a deadline forces a choice, label it an operational choice and retain the uncertainty in the experiment log.

## Continue reading
- /learn/conversion/test-prioritization/
- /learn/ios/product-page-optimization/
- /learn/google-play/store-listing-experiments/