Can the newest reviews tell a different story?

One app.
Two very different review snapshots.

AI Maker’s latest 200 retrieved reviews were 35% bad. Its newest 50 were 78% bad. Pooling the reviews into one number made the recent negative feedback look much less prominent.

Across 44 apps, this was the only change that met both our size and statistical requirements. This is a follow-up measurement of the pattern found in Experiment 20, not an independent discovery.

Only AI Maker passed both primary requirements. The proposed repeated pattern across at least three apps was not established. The other 43 apps had differences smaller than twenty percentage points under the main comparison.

Why this matters for ASO

An ASO audit might summarize “the last 200 reviews” and treat that as the current picture. But older records can dominate that number even when the newest written feedback looks very different.

We compared the newest 50 reviews with the 150 immediately before them. Those groups do not overlap. This separates a recent review snapshot from the earlier one instead of comparing the newest reviews with an average that already includes them.

Bad means a 1-star or 2-star written review. The percentage refers to the records retrieved, not all users or all App Store ratings. A shift from 20% to 40% bad is a twenty-percentage-point change.

AI Maker changed from 20.7% bad to 78% bad.

The difference between the separate blocks was 57.3 percentage points. A report using only the full 200-review figure would show 35% bad, even though more than three-quarters of the newest group were bad.

The broader number is not mathematically wrong. It answers a different question. Three-quarters of its records come from the older group, so it gives that group three times the weight of the newest one. That averaging effect is arithmetic, not separate evidence of a product change.

The time periods were different lengths

The older 150 records ran from 20 October at 06:09 UTC to 21 October at 05:00 UTC, roughly 23 hours. The newest 50 ran from 21 October at 05:01 UTC to 25 October at 21:55 UTC, roughly 4.7 days.

These are fixed numbers of retrieved reviews, not equal-length calendar windows. We cannot infer a change in complaints per day, per install or per user from these percentages alone.

How this relates to Experiment 20

The earlier experiment found an unusually long negative streak in this same app’s same 200-record sample. This experiment asks how much the selected review window changes the summary. It is a different calculation of an already observed pattern, not a second independent confirmation.

Five apps crossed the “mostly bad” line.

We called a block majority-bad only when more than half its records were bad, and minority-bad only when fewer than half were bad. Exactly 50% was kept separate. These five apps crossed between the two categories.

WhatsApp shows the risk of relying on the label alone. It moved from 48% bad to 52% bad. That changes the wording from minority-bad to majority-bad, but the underlying difference is only four points. In a 50-review group, one record represents two percentage points.

TikTok moved the other way: 58% bad in the older group versus 40% in the newest group. Its eighteen-point fall was below our twenty-point size threshold and did not pass the statistical requirement after correction for testing 44 apps.

PLAYFUL REWARDS and DoorDash also crossed the halfway line without meeting both requirements. A changed category is an observed fact; it does not automatically establish a large or persistent change in the app’s feedback.

Four other apps had exactly 50% bad in at least one block. They were not counted as majority flips. We did not round percentages before assigning labels.

Changing the window still left one qualifying app.

The newest-25 check produced more large percentage swings: five apps changed by at least twenty points. Yet only AI Maker also passed the corrected statistical test. A smaller group can show a large percentage swing with fewer reviews behind it.

No app had identical update timestamps across the primary split. The timestamp-exclusion check therefore changed nothing; it is not additional independent evidence.

AI Maker under the other definitions

All four checks retained AI Maker as the sole qualifying app. They reuse overlapping records and are not independent replications. Each statistical correction applies within its own comparison family.

Keep the comparison consistent across apps.

1. Start with the fixed 57-app US panel. Reuse the 21,750 written-review facts collected for Experiment 15 on 27 October 2026. No new reviews were collected for this experiment.
2. Require 200 retrieved records. Thirteen apps had fewer and were excluded. We did not lower their sample-size requirement or treat missing reviews as positive feedback.
3. Sort by UTC update time. Break timestamp ties by numeric review ID. Take the newest 200 records and split them into an older group of 150 and a newest group of 50.
4. Measure the change. Subtract the older bad-review percentage from the newest one. Keep the direction: higher or lower.
5. Require a change of at least twenty percentage points. This is the study’s chosen threshold, not an Apple standard.
6. Apply a two-sided Fisher exact test. Under a model with the same bad-review probability in both blocks, compare the observed table with all tables having the same totals. Correct for all 44 app comparisons using Holm’s method.
7. Flag only apps meeting both rules. The change must be at least twenty points and the corrected probability must be no greater than 0.05.

AI Maker’s raw p-value was about 4.92 × 10^-13; after correction it was about 2.16 × 10^-11. These probabilities describe the comparison model, not the probability that a particular cause is true.

The data exclusions

What limits the conclusion?

A recent block difference is not proof of a lasting change. Serial clustering can create differences between adjacent blocks. Fisher’s comparison assumes the bad/nonbad labels are exchangeable under the no-difference model. A stable process with dependent review bursts may violate that assumption. We cannot distinguish a temporary cluster from a sustained change in app experience here.

Dates can reflect edits. Apple allows customers to update their ratings and reviews, replacing earlier submissions. The sequence uses the retrieved records’ update times, which are not guaranteed to be original posting times.

The public feeds are incomplete historical evidence. Deleted reviews, pagination limits, delays and missing records can change the observed sequence. The original collection requested up to ten pages per app. Written reviewers are not all users, and star-only ratings are absent.

The panel and windows are selective. These are apps chosen from earlier US top-free chart appearances. Equal review counts span different periods and do not control for traffic, versions or user mix.

This is exploratory follow-on research. The rules were saved before computing this experiment’s results, but the data and AI Maker pattern had already been examined. Corrections apply within each test family, not across every question in the entire ASO research series.

This study does not demonstrate a ranking change, conversion change, app update effect, localization benefit or impact from editing a listing. The other 43 apps’ failure to meet the thresholds does not prove their feedback was stable or healthy.

Download the data and reproduce the calculation.

The evidence includes the original review facts and request logs, exact selected samples, all 57 app statuses, counts and dates, raw and corrected probabilities, all four alternative comparisons, the written rules and analysis code.

Verification rebuilt the review groups independently and checked every primary and sensitivity probability using a different combinatorial formula. It also checked 870 small tables against exhaustive binary assignments. All 69,582 processing checks passed. This validates the calculations, not feed completeness or the statistical assumptions.

Extract and run python analyse.py, then python verify.py. Both work offline using Python’s standard library. No reviewer names or review text are included.

1. Apple US written-review feed, AI Maker example. Original acquisition URLs, timestamps and response hashes are preserved in the download. Live feeds change.
2. Apple: ratings and reviews overview. Explains updated submissions.
3. R documentation: Fisher’s exact test. Reference for the two-sided table-probability definition.
4. R documentation: multiple-comparison correction. Reference for Holm’s method.
5. AppOrbit by ivershuo, pinned source. Earlier panel-selection source; derived-panel attribution and ODbL 1.0 license included.