Scott Alexander, curated
← Back to curation

The New AI Consciousness Paper

Quality
77
Excellent
Claude Shift
54
Moderate
RWI
2
of 10

Summary

Reviews the Bengio/Chalmers et al. paper 'Identifying Indicators of Consciousness in AI Systems,' which restricts itself to computational theories (Recurrent Processing, Global Workspace, Higher-Order - which Scott genially compresses to 'something something feedback') and concludes no current AI is conscious (LLMs are feedforward) but there's no technical barrier to building ones that satisfy the indicators. His incisive critique turns on the access-vs-phenomenal consciousness distinction: the paper commits to studying phenomenal consciousness (the 'what it's like,' qualia, moral value) but then 'cuts corners anyway' by reinterpreting access-consciousness theories as phenomenal ones on the assumption they coincide. 'Something something feedback' obviously tracks access consciousness (that's its definition) but not obviously the phenomenal kind - the p-zombie problem - and the theories-as-phenomenal yield strange reductios (Global Workspace implies a company of employees emailing a boss is conscious; IIT implies thermostats are, which Tononi accepted). Adds an aphantasia-like hypothesis about why people's intuitions on phenomenal consciousness differ so wildly, the Anthropic introspection result (AIs identify injected concepts above chance), and a strong section III prediction: we'll ascribe consciousness to AIs by use-case, not algorithm (humanlike boyfriend-bots vs deliberately-impersonal factory-bots, like dogs vs pigs with similar brains but opposite moral treatment).

Why this score

Quality 77 · Excellent. Excellent band (lower). Lucid on a famously hard topic, both a competent review of a real philosophy-of-mind paper and an incisive critique (the corner-cutting charge, the reductios), plus a genuinely insightful sociological prediction. Held at the lower Excellent band because it is anchored to the paper, leans on the standard access/phenomenal (Block) distinction, and Section I admits to glossing the technical theories. A=77.

Claude’s paradigm shift 54 · Moderate. Notable shift. The access/phenomenal distinction and the theories are not his; the fresh contributions are the aphantasia-of-consciousness hypothesis and the use-case-not-algorithm prediction, applied sharply to a live AI debate. B=54.

Real-world impact 2 · Minor. A lucid review and incisive critique of the Bengio/Chalmers AI-consciousness paper (the access-vs-phenomenal corner-cutting charge) plus a sociological prediction. Conceptual influence within AI/philosophy-of-mind discourse, anchored to the paper, no material change — low RWI.