Scott Alexander, curated
← Back to curation

The "Spot the Fakes" Test

Quality
71
Strong
Claude Shift
46
Moderate
RWI
2
of 10

Summary

Generalizes the Ern Malley hoax (two poets fooled a modernist magazine with random-phrase nonsense) into a proposed test for otherwise-untestable claims: submit one sincere argument plus several best-effort bogus ones, and have independent experts identify the real one; if they can't, or settle on a fake, the belief is probably wishful thinking. Worked through Max Muller's solar-deity mythology (a real Hercules argument hidden among fake Perseus/Bellerophon ones) and the Sokal affair, with a footnote on testing a smart Christian to sort ten Biblical atrocities from ten Bronze-Age-king ones. A clever, exportable epistemics tool (essentially the logic behind adversarial controls), but short and acknowledged to be fragile — it relies on the proposer's honesty, and both extended examples show it failing in practice.

Why this score

Quality 71 · Strong. Low-Strong. The 'spot the fakes' adversarial-control test is a genuinely clever, original idea that prefigures adversarial-collaboration methods, but the post is short and underdeveloped and mostly demonstrates how hard the test is to apply, capping it at the Solid/Strong boundary.

Claude’s paradigm shift 46 · Moderate. Notable — turning the Sokal/Ern Malley one-off hoax into a repeatable blinded test for untestable fields was a reasonably novel methodological move in 2012.

Real-world impact 2 · Minor. Generalizes the Ern Malley hoax into a clever adversarial-control test (hide one sincere argument among best-effort fakes; if experts can't pick the real one, it's wishful thinking) that prefigures adversarial-collaboration methods. Conceptual influence within rationalist discourse, short and underdeveloped, no material change — low RWI.