A Machine Alignment Monday explainer of MIRI's 'Problem of Fully-Updated Deference' critique of CHAI's (Stuart Russell's) assistance-games / inverse-reinforcement-learning alignment agenda. Scott makes a dense debate genuinely accessible: the opening blue/red-paperclip skit; the sovereign-vs-corrigible distinction; the moral-uncertainty 'utility function in an envelope' approach; why IRL is hard for modern gradient-descent AIs (we don't know how to point them at real-world referents at all — Eliezer's 'maximize staples if snow is purple' point) and why even solving that leaves the problem that humans don't have clean utility functions (so someone must hard-code the meta-rules correctly the first time); CHAI's corrigibility argument (the AI lets itself be shut off if human/AI utility functions are correlated enough); and MIRI's rebuttal (the AI's 6th option — refuse shutdown, gather info, then optimize its true, subtly-wrong function). Hosts an actual Eliezer-vs-Stuart-Russell exchange and fairly identifies the crux (whether the AI can end up with an uncorrectably-wrong model of human values).

Quality 76 · Excellent. Strong, near-Excellent: lucid, well-structured pedagogy that makes a hard, central alignment argument accessible (the skit-based explanation is genuinely clever) and hosts the real expert exchange; a standout among Scott's alignment explainers (with Yudkowsky-Contra-Christiano). Held just below Excellent as an explainer of others' debate.

Claude’s paradigm shift 56 · Moderate. Major-ish: transmits and clarifies the fully-updated-deference / corrigibility problem — a central, load-bearing alignment result — for a broad audience.

Real-world impact 3 · Moderate. Moderate (3): an influential explainer of a core alignment problem (corrigibility / fully-updated deference) within the AI-safety field.

CHAI, Assistance Games, And Fully-Updated Deference