Open Questions On Open Weights
Read the original on Astral Codex Ten →
Summary
Response to the pro-open-weights open letter (Microsoft, NVIDIA, OpenAI, Meta, Hugging Face and others). Scott explains why the AI-safety 'conspiracy' is mostly neutral and states his own neutrality. Takeover risk from open weights is bounded by the ~6-month closed-weights lead. Misuse risks (hacking, bioterror) fail what he calls the foreshock test: the body politic reacts to disasters rather than preparing for them, so you should spend political capital on pre-emptive action only where waiting for the first foreshock would be too late. Superintelligent takeover passes that test because a smart adversary hides until ready. Criminal hacking and mostly incompetent bioterrorism (median deaths zero) do not, because a 2x-uplift incident will trigger bans long before a 1000x one. He acknowledges the rare-event counterargument and grants that open weights' 'free yeomen vs corporate serfs' case deserves its chance.
Why this score
Quality 65 · Strong. Mid Strong. The political-capital and foreshock framework is a genuinely useful strategic heuristic, applied with honest caveats, and it clarifies where the safety movement should and shouldn't fight. But it is short, rests on contestable empirical bets (the closed-open lag, bioterror incompetence), and is partly positioning. Med confidence for recency.
Claude’s paradigm shift 46 · Moderate. Moderate. 'Societies react rather than prepare' is old; using it as a triage rule for where to spend preemptive political capital is a modest fresh synthesis.
Real-world impact 1 · Negligible. 1: articulates a stance within AI-policy discourse; no observed effect.