THE VERDICT · NOT SUPPORTED
One of 23 comparisons showed a clear change in complaint topics.
In my selected sample, one of 23 app comparisons passed both of the study’s rules for a large, statistically detectable change in issue words. My original prediction required at least six.
The result does not mean the other apps had no changes. The review samples and methods could leave genuine differences undetected.
See what I found ↓
MY QUESTION
Am I reading complaints from an old version?
If I use old reviews to decide what an app listing should explain, I might focus on a problem customers mentioned years ago while missing what newer reviews discuss. I wanted to know whether the topics in selected low rating reviews changed between versions.
I matched historical Google Play review records to release versions for the same app.
I compared the earliest and latest qualifying versions, each with at least 30 eligible reviews.
I checked five word groups, including crashes, slow performance, accounts, payments and ads.
These are word matches in selected one and two star reviews. They are not a validated measure of every complaint or an app’s reliability.
MY THEORY VS THE RESULT
My broad prediction failed.
My theory
At least one in four eligible apps would show a change of at least 10 percentage points in an issue topic and pass the study’s statistical check.
What I found
Only one of 23 comparisons passed both rules. I needed at least six of 23 for my prediction to pass.
23 app comparisons → apply both rules
Passing required both a large enough difference and the study’s adjusted statistical check.
See the original result chart
Each app contributed one comparison between two qualifying versions.
Passing required a difference of at least 10 percentage points and a statistical result that accounted for testing multiple apps. The rules were chosen for this study, not an industry standard.
Nine comparisons had a difference of at least 10 points, but only one also passed the statistical check. A result that did not pass is not proof that nothing changed.
REAL APPS IN MY SAMPLE
What did the reviews show?
These are historical Google Play records from selected open source Android apps. The source extract does not establish a reliable country for these examples. They are not claims about today’s app quality, developer intent or client relationships.
GOOGLE PLAY · TWO SELECTED VERSIONS
Kodi
Words about crashes or freezing appeared in 21 of 101 selected reviews labelled version 18.1 and 28 of 467 labelled version 19.0. That is 20.8% versus 6.0% within those selected reviews.
It does not mean the newer app was 71% more reliable. I measured the share of selected reviews mentioning those words, not crashes per session.
GOOGLE PLAY · TWO SELECTED VERSIONS
Barcode Scanner
Ad related words appeared in 1.3% of selected reviews for version 4.7.3 and 10.6% for version 4.7.8. That difference was 9.3 percentage points, below the study’s chosen 10 point threshold.
GOOGLE PLAY · TWO SELECTED VERSIONS
SMS Backup+
Account related words rose from 0% to 15.2% in the selected reviews. This exceeded the size threshold, but did not pass the adjusted statistical threshold. I did not count it as a success.
Kodi was selected after examining all 23 comparisons. It is an example supported within this dataset, not evidence that I predicted the specific result in advance.
MY ASO TAKEAWAY
Check when a review was written and which version it describes.
My ASO tip: Before using a complaint to shape listing copy, check whether it describes an older version and whether the issue still applies.
My ASO strategy: Group reviews by version and date, then inspect the actual text. Route product problems to the product or support team and verify features before changing the listing.
My ASO warning: A change in the mix of review topics does not prove an update fixed a problem or that changing screenshots or copy will improve downloads.
WHAT I COULD NOT PROVE
Here is where the evidence stops.
- Cause: Different users, devices, services and expectations could influence the reviews. I did not isolate the effect of an update.
- Reliability: A share of reviews mentioning crashes is not the chance of a crash for an app user.
- Every complaint: My rules count selected words. They miss other issues and can match a word used in an unrelated way.
- Version labels: I found a review whose text discussed a different version from its recorded label. That warning may apply beyond the one example inspected.
- Today’s market: The archive ends in 2023 and covers 23 selected open source Android apps, not every app or user.
SOURCES & METHODS
Want to check my work?
I used the DATAR archive and matched selected reviews to recorded releases. I reran the supplied recount against the saved records and source manifest. The checks verify the selection and arithmetic, not the accuracy of the language screening, review labels or causal explanations.
My comparison used 7,117 selected reviews across 23 apps and 46 versions. One review can match more than one topic. The source evidence and full technical report are kept locally while third party redistribution rights are reviewed.
Read the original report and technical details
The clearest example: Kodi
Among the selected one- and two-star reviews, crash and freezing words appeared in 20.8% of reviews labeled version 18.1 and 6.0% labeled version 19.0.
Share of low-star reviews mentioning crashes or freezing20.8%Kodi 18.121 of 101 eligible reviews6.0%Kodi 19.0
28 of 467 eligible reviews
A difference of 14.8 percentage points. Imagine reading 100 low-star reviews from each group: roughly 21 would mention crashes or freezing in the older group, compared with 6 in the newer group. These are rounded rates, not two samples of exactly 100.
That does not mean Kodi became 71% more reliable. We measured what people wrote in low-star reviews, not crashes per app session. Other complaints can grow and make crash complaints a smaller share. In fact, the newer sample contains more crash mentions in raw numbers because it has many more reviews.
The Kodi difference remained above our threshold in all six planned checks. Those checks include looking only at one-star reviews and only at reviews posted within 90 days of the recorded GitHub release.
Why this matters for ASO
ASO means app store optimization. When an ASO team reads bad reviews to understand users' objections, mixing years of app versions can hide which problems belong to which period.
For example, suppose your app's old reviews often mention crashes, but newer reviews talk about missing features. A store-page explanation about stability may address an old concern while leaving the newer concern unanswered. This is an illustrative ASO decision, not an outcome we tested.
The practical lesson from this case: check the version and date before using old reviews to decide what an app listing should explain. This study does not show that changing listing text improves downloads, ratings, or rankings.
The theory we actually tested
Our prediction: at least one in four eligible apps would show a change of 10 percentage points or more in one of five issue topics, with enough evidence to pass our statistical check.
For example, moving from 10 out of 100 reviews mentioning login problems to 25 out of 100 is a 15-point change. Moving from 10 to 12 is only a 2-point change.
What happened: 9 of the 23 app comparisons had a change of at least 10 points. After also accounting for sample size and the many comparisons, only 1 of 23 passed both rules. We needed at least 6 of 23 to meet the original prediction.
We did not prove a general ASO rule. The broad prediction failed this test. Kodi is a supported example within this dataset, selected after examining all 23 comparisons. It is not a replacement hypothesis that we can pretend was predicted in advance.The observed pass rate was 4.3%. Its descriptive 95% interval was 0.8% to 21.0%. This interval describes uncertainty in this selected sample; these apps are not a random sample of the app stores.
What we counted
We used archived Google Play reviews from DATAR, a research dataset of open-source Android apps. After the matching and language checks, the main comparison contained 7,117 reviews across 23 apps and 46 app versions. The selected review dates run from August 2014 to December 2023.
- Match the same app. Connect a review's recorded version to the dataset's release version exactly. Exclude ambiguous package mappings and draft or prerelease records.
- Keep low-star reviews with enough text. Use one- and two-star reviews, at least 20 characters and four alphabetic words, classified as English by a language tool.
- Choose versions before comparing topics. Each version needs at least 30 eligible reviews. For each app, compare its earliest and latest qualifying versions by GitHub publication date.
- Count five kinds of words. Crashes/freezing; slow performance; login/account access; payments/subscriptions; advertisements. One review can count in several groups.
- Test every app by the same rules. Require both a 10-point change and a statistical result that survives adjustment for testing multiple apps.
These are issue-word mentions, not a fully validated complaint classifier. Only 1,066 of the 7,117 reviews, or 15.0%, contained one of the selected terms. Many other complaints concern backups, scanning, add-ons, usability, or words our rules missed.
What the review text revealed
We read a fixed random sample of 90 reviews and all 49 Kodi reviews matching crash/freezing terms. The examples below are paraphrases, not quotations.
| Example from the text | What it teaches us |
|---|---|
| A Kodi reviewer describes the app freezing during startup. | A relevant crash/freezing complaint. |
| A reviewer says they would pay for an app if it worked again. | A payment word does not necessarily mean a billing complaint. |
| A reviewer says code was changed slowly. | “Slowly” does not necessarily describe app speed. |
| A review labeled Kodi 18.1 discusses version 20 and a rollback to 19.3. | A recorded version can disagree with edited review text. |
Removing that one observed Kodi version mismatch changes the older figure to 20.0% (20 of 100), compared with 6.0% (28 of 467) for 19.0. This extra check was done after seeing the results. It is not a complete correction of all possible version errors.
The text review was performed by the same assistant that ran the analysis. It was a diagnostic check, not independent human validation. We kept the original counting rules and disclosed their weaknesses.
All 23 app comparisons
The table shows the topic with the largest change for each app. “Pass” means the change reached 10 points and survived the multiple-comparison check. A “No” does not prove there was no change. Small review samples often leave uncertainty.
Review counts are older / newer. Change is in percentage points. The adjusted p-value is included for readers who want to audit the statistics.
| App | Versions | Reviews | Largest-changing topic | Older → newer | Change | Adjusted p | Pass? |
|---|---|---|---|---|---|---|---|
| Kodixbmc/xbmc | 18.1 → 19.0 | 101 / 467 | Crash or freezing | 20.8% → 6.0% | -14.8 | 0.0011 | Yes |
| Barcode Scanner (ZXing)zxing/zxing | 4.7.3 → 4.7.8 | 313 / 417 | Advertisements | 1.3% → 10.6% | +9.3 | 0.0011 | No |
| SMS Backup+jberkel/sms-backup-plus | 1.5.9 → 1.5.11 | 47 / 1,825 | Login or account access | 0.0% → 15.2% | +15.2 | 0.0529 | No |
| DuckDuckGoduckduckgo/Android | 5.164.2 → 5.178.0 | 56 / 43 | Crash or freezing | 5.4% → 23.3% | +17.9 | 0.0949 | No |
| Mastodonmastodon/mastodon-android | 1.1.3 → 1.1.4 | 375 / 87 | Login or account access | 22.4% → 11.5% | -10.9 | 0.1168 | No |
| kiwix-androidkiwix/kiwix-android | 2.4 → 3.4.5 | 30 / 38 | Crash or freezing | 33.3% → 10.5% | -22.8 | 0.1248 | No |
| lbry-androidlbryio/lbry-android | 0.17.0 → 0.17.1 | 37 / 151 | Crash or freezing | 16.2% → 33.8% | +17.6 | 0.1248 | No |
| slssimple-last-fm-scrobbler/sls | 1.4.8 → 1.7.0 | 59 / 75 | Login or account access | 1.7% → 12.0% | +10.3 | 0.1248 | No |
| ContentBlockerAdguardTeam/ContentBlocker | 2.1.2 → 2.6.0 | 42 / 47 | Advertisements | 35.7% → 55.3% | +19.6 | 0.1650 | No |
| ActivityLauncherbutzist/ActivityLauncher | 1.9.1 → 1.14.4 | 45 / 55 | Crash or freezing | 6.7% → 0.0% | -6.7 | 0.1650 | No |
| snapdrop-androidfm-sys/snapdrop-android | 1.11.0 → 1.11.3 | 31 / 44 | Slow performance | 9.7% → 0.0% | -9.7 | 0.1650 | No |
| lichobilelichess-org/lichobile | 6.1.1 → 8.0.0 | 50 / 376 | Slow performance | 2.0% → 10.9% | +8.9 | 0.1650 | No |
| WhereYouGocgeo/WhereYouGo | 0.9.3 → 2022.01.14 | 33 / 37 | Login or account access | 15.2% → 2.7% | -12.4 | 0.3883 | No |
| UniPatcherbtimofeev/UniPatcher | 0.16.1 → 0.17.2 | 36 / 67 | Crash or freezing | 0.0% → 6.0% | +6.0 | 0.4802 | No |
| AcDisplayAChep/AcDisplay | 3.7.1 → 3.8.4 | 32 / 663 | Slow performance | 6.2% → 1.4% | -4.9 | 0.6185 | No |
| ppsspphrydgard/ppsspp | 1.5.4 → 1.14.4 | 145 / 118 | Crash or freezing | 10.3% → 14.4% | +4.1 | 0.6303 | No |
| shattered-pixel-dungeon00-Evan/shattered-pixel-dungeon | 1.3.2 → 1.4.3 | 39 / 32 | Payments or subscriptions | 0.0% → 3.1% | +3.1 | 0.8999 | No |
| uhabitsiSoron/uhabits | 1.5.6 → 2.0.3 | 33 / 55 | Crash or freezing | 3.0% → 0.0% | -3.0 | 0.8999 | No |
| qksmsmoezbhatti/qksms | 3.8.1 → 3.9.2 | 77 / 144 | Slow performance | 0.0% → 2.8% | +2.8 | 0.8999 | No |
| oxygen-updateroxygen-updater/oxygen-updater | 5.3.0 → 5.5.0 | 33 / 32 | Payments or subscriptions | 0.0% → 6.2% | +6.2 | 0.8999 | No |
| Launcher3amirzaidi/Launcher3 | 3.8.1 → 3.9.1 | 37 / 118 | Crash or freezing | 5.4% → 8.5% | +3.1 | 0.9716 | No |
| quran_androidquran/quran_android | 2.7.1 → 3.2.0 | 36 / 36 | Crash or freezing | 0.0% → 2.8% | +2.8 | 1.0000 | No |
| stardroidsky-map-team/stardroid | 1.8.2 → 1.9.2 | 34 / 469 | Crash or freezing | 2.9% → 4.1% | +1.1 | 1.0000 | No |
Barcode Scanner's ad-word change passed the statistical check but measured 9.3 points, below our chosen 10-point rule. SMS Backup+ had a 15.2-point login-word change but narrowly missed the adjusted statistical threshold. We did not relax either rule to count them as primary successes. Reviews can contain mistaken allegations; topic counts do not verify those allegations.
Did the result depend on one choice?
We repeated the analysis using six checks specified before the topic results. None produced a pass rate of one in four. The Kodi example remained above the full threshold each time.
| Check | Apps passing | Reviews | Kodi still passes? |
|---|---|---|---|
| English confidence at least 0.95 | 1 of 23 | 7,022 | Yes |
| One-star reviews only | 2 of 20 | 5,025 | Yes |
| At least 50 reviews per version | 1 of 7 | 2,804 | Yes |
| Within 90 days of GitHub release | 1 of 10 | 1,432 | Yes |
| Latest two eligible versions | 2 of 23 | 7,461 | Yes |
| Exclude largest app pair | 1 of 22 | 5,245 | Yes |
Restricted subsets need at least 20 reviews per version, except where the table specifies 50. Choosing the latest two versions uses the original 30-review minimum. Each check adjusts its own set of comparisons. The 90-day check uses GitHub dates, which are not verified Google Play rollout dates.
What this study cannot tell us
- Whether the app update caused the difference. Reviewers, devices, operating systems, backend services, and expectations can change between versions.
- Whether app quality improved. We measured the mixture of topics inside selected low-star reviews, not the chance that a user has a problem.
- Whether an ASO change would help. There are no listing experiments, conversion measurements, download outcomes, or chart rankings here.
- Whether the pattern applies to most apps. The sample contains only 23 eligible open-source Android apps. It excludes many short, non-English, unmatched, or sparsely reviewed versions.
- What users think today. The archive ends in 2023. The newer version in a pair is only the latest one qualifying in this archive.
- Whether all version labels are correct. We observed a label/text mismatch. Some reviews were posted long after the recorded release, and the two groups are not the same people.
Publication-ready claim: In this historical sample, Kodi reviews assigned to versions 18.1 and 19.0 differed substantially in the share mentioning crashes or freezing. The same strict test did not establish that such large, detectable shifts were common across the 23 eligible apps.
Open the evidence
The download includes the fixed analysis plan, selected review records, exact word rules, all app results, sensitivity checks, Python analysis, and verification code. 216 independent recount and integrity checks passed. This verification checks the calculations and files; it does not validate the language classifier or causal claims.
Technical methods, in plain language
For each app, we take the largest absolute difference across the five topic rates. We then randomly reassign the observed five-topic combinations between the two version groups 9,999 times, keeping the number of reviews in each group unchanged. This asks how unusual the observed difference is if the groups have the same distribution of these topic indicators.
The permutation keeps topics that occur together in the same review together. It uses an add-one p-value and a fixed random seed, 13027. We apply Benjamini-Hochberg adjustment across the 23 apps and require an adjusted p-value below 0.05. This helps limit false discoveries when many apps are tested. It does not remove confounding, text errors, or dependence between reviews.
Each app counts once toward the overall pass rate. The confidence interval is Wilson's 95% interval. The 10-point and 25% thresholds are practical choices made for this study, not accepted ASO standards. The protocol was frozen locally after checking data availability and before calculating topic outcomes; it was not registered with an external service.
English screening uses langid 1.1.6 with normalized confidence at least 0.8, and a stricter 0.95 check. Its confidence is a tool score, not a guarantee of correct language. Exact regular expressions are provided in patterns.json. We did not manually relabel the full dataset.
How the source became this sample
| Reviews in original extract | 1,335,331 |
|---|---|
| Release records | 8,041 |
| Matched low-star reviews after package/release checks | 25,603 |
| Candidate reviews in versions with at least 30 low-star reviews, in apps having at least two such versions | 16,192 in 31 apps |
| Candidate reviews passing English/text screening | 12,328 |
| Versions retaining at least 30 eligible reviews | 95 |
| Earliest/latest eligible pairs | 46 versions in 23 apps |
| Reviews used in the main comparison | 7,117 |
The 95 eligible versions include some apps left with only one qualifying version. Those apps cannot enter the paired comparison.
Sources and prior research
DATAR by Yasaman Abedini, Mohammad Hadi Hajihosseini, and Abbas Heydarnoori: archived dataset, CC BY 4.0; project documentation; research paper. The evidence contains hashes identifying the exact extracts used.
Version-specific review analysis is an existing research idea. Yang and colleagues' TOUR: Dynamic Topic and Sentiment Analysis of User Reviews for Assisting App Release studies changing topics and sentiment across app versions. Our contribution here is a reproducible, narrowly defined comparison on this archived sample, with both positive and negative results disclosed.
Language screening: langid.py official repository. All percentages in the findings are calculations from the supplied study data, not estimates taken from those papers.
Keep exploring
All ASO experiments · See my positive review study · Track listing changes · Clarify your app’s value