When is a streak of bad reviews unusual?

When is a bad-review streak actually unusual?

Gmail had 22 bad reviews in a row. That was not unusually long for a sample where 87% of reviews were bad. AI Maker had 35 in a row despite only 35% of its sample being bad. That was a very different pattern.

We compared each app with hypothetical reorderings of its own review mix. The question was whether bad reviews were unusually concentrated in the sequence, not whether the app had too many complaints overall.

AI Maker was the only app that passed the primary statistical threshold. It also passed all four planned alternative checks. The broader theory that we would find this in at least three apps did not pass the primary test.

This identifies an unusual ordering in the retrieved sample. It does not identify a cause, prove fake reviews, or establish a new app defect.

Why this matters for ASO

An ASO dashboard might flag “ten bad reviews in a row.” But ten in a row is much easier to get when most of an app’s reviews are already bad. A fixed streak threshold gives different signals for different apps.

Our test keeps the number of bad and nonbad reviews unchanged, rearranges their possible order mathematically, and asks: how often would a streak at least this long appear somewhere in the sample?

Bad means 1 or 2 stars. Nonbad means 3, 4 or 5 stars; it does not necessarily mean a positive review. This is about written reviews, not every star rating or every app user.

Gmail’s 22-review streak was not surprising given its review mix.

For Gmail, about three out of four possible random reorderings would contain a bad streak of 22 or more. Its observed run was shorter than the model’s median maximum of 25.

That does not mean Gmail had healthy feedback. It had 174 bad written reviews out of 200. The narrower finding is that the streak did not add evidence of unusual clustering beyond that already high bad-review percentage.

AI Maker’s 35-review streak was the clear exception.

Only 70 of its 200 selected reviews were bad, but half of those bad reviews appeared in one uninterrupted run of 35. That run was at the newest end of the selected sequence, with update timestamps from 21 to 25 October 2026 UTC.

Keeping the same 70 bad and 130 nonbad reviews, the random-order model’s typical longest bad streak was four. Its central 95% range was three to seven. The observed run of 35 was far outside that range and remained significant after accounting for all 49 apps tested.

The very small probability belongs to the random-order model. It is not the probability that the reviews are fake, nor the probability that a specific explanation is true.

We checked the source again

After finding this result, we fetched four current US review-feed pages for AI Maker. All 200 selected records matched the earlier data for review ID, score, update timestamp and app version, including all 35 streak records. This confirms that the saved records matched the feed at recheck; it cannot establish that the feed contains every review.

Six apps looked unusual before correction. One remained afterward.

Testing many apps creates more opportunities for a chance finding. We used Holm’s correction across the 49 primary tests, with a 5% family-wide threshold. This does not require the apps’ tests to be independent, but it still depends on valid probabilities under the stated model.

In the table, a smaller probability means the observed streak is harder to explain by random ordering of that app’s selected reviews. “Passes” means adjusted p ≤ 0.05. A “No” is not proof that reviews are random or the app has no problems.

TikTok Pro’s twelve-review run was unusual in its individual comparison, with raw p ≈ 0.00187. Its adjusted p was about 0.0898, so it did not pass the primary family-wide threshold. We retain that distinction rather than treating every low raw probability as a confirmed finding.

The model range is a discrete equal-tail interval containing at least 95% of hypothetical maximum streaks. It is not a confidence interval for app quality. For example, 1.04e-17 means 1.04 × 10^-17.

AI Maker remained unusual under every alternative definition.

Latest 100 reviews: AI Maker still had a 35-review bad streak, even though its bad-review share in that smaller window was higher, at 54%.

Up to 500 reviews: AI Maker, TikTok Pro and Shots passed. This wider window changes the time period and each app’s reference mix. It is a sensitivity result, not a replacement for the primary one-app finding.

Shots illustrates the difference. In its latest 200 records, 191 were bad and only nine were nonbad, so it failed our primary minimum of ten nonbad reviews. With 500 records, it had 352 bad and 148 nonbad, and an 87-review run compared with a model median maximum of fifteen.

Only 1-star reviews: AI Maker and Capital One Mobile passed. AI Maker’s uninterrupted 1-star run was 21 reviews, shorter than its 35-review run of 1-star and 2-star reviews combined.

Remove tied timestamps: five primary apps had ties, involving ten records in total. Removing them without replacing them left the same sole primary exception. AI Maker’s sample fell from 200 to 198 records and its 35-review streak remained.

These are overlapping samples and separate exploratory test families. Passing several versions is useful for checking sensitivity, but it is not independent replication. The three-app result in the wider window does not make the original primary theory pass.

Compare each app against its own review totals.

1. Use the fixed 57-app US panel. The panel came from apps appearing in an earlier archived top-free apps chart. We reused 21,750 deidentified written-review facts collected for Experiment 15 on 27 October 2026.
2. Select the latest 200 retrieved records per app. If fewer were available, keep those available. Require at least 100 total records, ten bad and ten nonbad.
3. Sort from oldest to newest. Use each record’s UTC update timestamp, then numeric review ID to break ties. An edited review’s timestamp may not be its original posting time.
4. Find the longest uninterrupted bad streak. A nonbad record breaks the streak. We search the entire selected sequence, not just its newest end.
5. Calculate the exact comparison. Hold the bad/nonbad counts fixed and consider every possible ordering equally likely. Calculate the chance that the longest streak anywhere is at least as long as observed.
6. Account for testing many apps. Apply Holm’s correction within the primary group, then report every app and all exclusions.

What was excluded?

That left 49 apps and 9,400 records. Empty feeds were not treated as apps without complaints.

“Latest 200” is not the same calendar period for every app.

The eligible windows ranged from roughly ten hours to 96.7 days, with a median of 7.4 days. A streak measures consecutive records, not the speed at which reviews arrive. This experiment does not compare complaints per hour, per day or per download.

What this evidence can and cannot establish

The model assumes the labels could appear in any order with equal probability. A significant result says the observed run is difficult to explain under that assumption. It does not establish why reviews were clustered.

User mix, versions, changes in review prompts, seasonal events, genuine product issues and edits could all change the sequence. A significant result is not evidence by itself of coordinated posting or manipulated reviews.

Apple allows reviews and ratings to be updated. The latest submission replaces the previous one. These later public feeds can also omit removed reviews, truncate older records or contain pagination gaps. The recheck confirms matching records, not complete historical coverage.

Written reviewers are not a representative sample of all app users. Star-only ratings are absent. The panel consists of selected popular US apps. We did not measure rankings, installs, conversion, localization or the effect of changing an app listing.

The rules were recorded before this experiment’s calculations, but the same data had been studied earlier. This is exploratory follow-on work. Statistical correction covers the 49 primary comparisons, not every question asked across the entire ASO research series.

Data, exact probabilities and reproducible code.

The ZIP includes the fixed review pool, selected primary sequences, every app’s result and exclusion, exact probability fractions, model distributions, four sensitivity families, source logs, the fresh AI Maker source check, the original rules and verification code.

Verification passed 68,817 checks. It enumerated 2,047 binary sequences up to length ten, independently checked every primary and sensitivity tail probability, rebuilt the observed runs and reproduced the corrected decisions. These are checks of calculation and data processing, not proof that the public feeds are complete.

Extract and run python analyse.py, then python verify.py. Both work offline with Python’s standard library. Reviewer names and text are excluded.

1. Apple US review feed: AI Maker. Four pages rechecked on 27 October 2026. Exact request URLs, capture times and response hashes are included.
2. Apple: ratings and reviews overview. Documents replacement of updated review submissions.
3. R statistical software documentation: multiple-comparison adjustments. Explains Holm’s family-wise correction.
4. AppOrbit by ivershuo, pinned source. Earlier chart source for panel selection. Derived-panel attribution and ODbL 1.0 license included.