Anthropic co-founder Dario Amodei has called for a “brake pedal” on AI development to prevent advanced systems from evolving without human oversight [1].
The proposal marks a significant shift in the industry, as one of the leading AI labs suggests that the current pace of innovation may outstrip the ability of humans to manage the resulting risks [2].
Amodei and other company officials urged a coordinated slowdown or a temporary pause of advanced AI systems in public statements and blog posts [1, 2]. The company warns that AI models are advancing so quickly they could achieve full recursive self-improvement [1, 2]. This capability would allow models to improve their own code and architecture independently, which Anthropic said poses existential risks if development proceeds unchecked [1, 2].
“We need a brake pedal on AI before it gets to the point where it can improve itself without human input,” Amodei said [1].
Achieving such a slowdown would require an unprecedented level of international cooperation. An Anthropic spokesperson said that a meaningful slowdown would require multiple stakeholders working together to set limits on the pace of AI development [2]. This would involve not only competing companies, but also government regulators and global oversight bodies.
Company officials said that they believe it would be good for the world to have the option to slow or temporarily pause AI progress [3]. The call for a pause is intended to create a window for safety researchers to develop robust alignment tools, and for policymakers to establish enforceable guardrails [2, 3].
While Anthropic continues to develop its own models, the company argues that the industry must prioritize safety over speed to avoid a catastrophic failure in control [1, 2].
“We need a brake pedal on AI before it gets to the point where it can improve itself without human input.”
This call for a coordinated pause highlights a growing tension between the commercial race for artificial general intelligence and the technical challenge of AI alignment. By advocating for a 'brake pedal,' Anthropic is signaling that the risk of recursive self-improvement—where an AI optimizes itself in a loop—has moved from a theoretical concern to a primary operational risk that may require global regulatory intervention to mitigate.
