Scott Alexander, curated
← Back to curation

Never Cross a River Four Feet Deep on Average

Quality
72
Strong
Claude Shift
48
Moderate
RWI
2
of 10

Summary

Guest post by 2024 ACX grantee Sasha Putilin reporting a replication of a 2023 study claiming individualized alpha-frequency light entrainment speeds perceptual learning. It failed to replicate; the original's headline 3x effect was an artifact of a few bored P-match participants with large negative learning rates, obscured by group-level averaging ('cargo-cult statistics'). Fully preregistered, open code/data, double-blind, with an honest appendix. Closes with a 'democratic meta-science with AI' thesis -- amateurs can now re-audit papers cheaply, so science may become 'more democratic than prestigious.'

Why this score

Quality 72 · Strong. Strong: a rigorous, transparent, well-explained replication whose real lesson (aggregate statistics hiding fragile individual-level effects) is delivered concretely, plus a thought-provoking AI-meta-science coda. Held below the top of Strong by guest-level prose and a fairly niche target; the methodological point is standard replication-crisis fare done well.

Claude’s paradigm shift 48 · Moderate. Moderate-low: neither the 'averaging hides fragility' critique nor the amateur-audit vision is novel in itself; the value is the clean worked example, not a new frame.

Real-world impact 2 · Minor. Minor/within-discourse: one replication of a niche study; models an approach (cheap amateur audits) with some community-discourse pull but no concrete material change.