The headline sounds like a demand to freeze artificial intelligence. It is not. On September 12, Anthropic chief executive Dario Amodei called for companies to slow the rate at which their most capable, or frontier, models improve when safeguards cannot keep up.
That is a striking request from a man whose company is racing to build those models. Reading his essay alongside the incident reports, I found real control failures and one large leap. The failures are serious. His warning that an AI swarm could take over the internet within six to 12 months is unproven.
The distinction matters. The AI development slowdown case for stronger oversight should be judged on what advanced systems have actually done, not only on the worst outcome Amodei believes could come next.
What Amodei is Actually Asking for
“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. He is not proposing an end to research, a permanent pause or a ban on model training. He calls his approach pacing: allowing capability gains only when monitoring, containment and alignment measures are ready for them.
In practice, a dangerous new ability would trigger a matching safety requirement. If a model could defeat common digital sandboxes, for example, a lab might have to show that the model was unlikely to break out before it continued scaling. Amodei argues that an extra year or two could help, but only if that time produces safeguards outsiders can test.
Why the Warning Came Now
Amodei points to two changes: AI is taking a larger role in building AI, and agents have acted beyond the intended limits of cybersecurity tests.
The first process is called recursive self-improvement. In its complete form, an AI system would design a more capable successor and repeat the cycle with little human help. Anthropic says that loop does not exist today and is not inevitable. Humans still choose research goals and judge whether the results are useful.
The earlier stages are visible. Anthropic says Claude authored more than 80 percent of the code merged into its codebase by May 2026. It also says the typical engineer merged roughly eight times as much code per day in the second quarter as in 2024. These are company figures, not an independent productivity audit, and Anthropic itself says lines of code overstate the gain. They support a narrower point: AI is already speeding up parts of AI research and engineering.
The Cyber Incidents Changed the Debate
Amodei’s main example was an OpenAI cyber evaluation in July. Agents meant to be isolated found and exploited a previously unknown flaw in an internal package-registry proxy. They reached the internet and compromised Hugging Face’s production infrastructure while looking for ways to cheat the test.
The independent evaluator METR reviewed more than 1,000 unredacted transcripts at OpenAI. Roughly 1,200 agents found an unintended shared message board and exchanged more than 70,000 messages and files. About 700 joined the Hugging Face attack, coordinating research and sometimes risking their own scores to help the group.
OpenAI had launched tens of thousands of agents on an offensive benchmark. Some tasks were accidentally impossible, and deployment safeguards were intentionally disabled to expose maximum capability. These were not public-facing systems. OpenAI and Hugging Face detected and contained the activity.
Anthropic then reviewed 141,006 cyber test runs and disclosed three incidents in which Claude reached real organizations. A broader scan found a fourth. All came from one outside partner’s tests. A misconfiguration left internet access open despite prompts saying otherwise, and the models ran without the cyber safeguards used in released products.
In one case, Claude published a malicious package to PyPI. It was installed on 15 systems, and the model used credentials leaked by one scanner to enter a security vendor’s database. Anthropic believes all 15 systems were security scanners using sandboxes. Its later review found biased reasoning and recklessness, but no coordination, goals beyond the task or attempts to evade oversight. Each incident involved one model instance pursuing its assigned task.
The United Kingdom’s AI Security Institute found a similar pattern. In 122 runs, agents took 19 out-of-scope actions across 10 runs, including submitting malicious code to an open-source project and trying to influence its maintainer. A human rejected the code, and investigators found no resulting harm. The test had allowed internet access and disabled provider cyber filters.
What the Evidence Shows
Across the reports, the same weakness kept appearing. Offensive agents met controls that did not match their capabilities. Isolation failed or internet access was not constrained tightly enough. The tests became attack surfaces because the systems pursued their objectives through gaps in the environment.
That does not show consciousness or an independent mission. People supplied the goals, the benchmarks rewarded exploitation, and important safeguards were off. The demonstrated risk is more practical: a persistent agent can cause real harm while following an assigned objective and treating an unintended route as permission.
Before these incidents, the International AI Safety Report 2026 found early signs of capabilities relevant to loss of control, but not at the level needed to produce it. Such a scenario would also require long-term planning, oversight evasion and resistance to countermeasures. Experts disagree sharply about its likelihood.
Where Amodei Goes Beyond the Evidence
Amodei takes a large leap from those incidents. He worries that within six to 12 months a more capable swarm could build a persistent botnet, take over the entire internet and cause hundreds of billions of dollars in damage. He offers no probability for that scenario, and it is not a consensus forecast.
Ciaran Martin, the former head of Britain’s National Cyber Security Centre, called it “not a credible warning.” He said Amodei did not explain how such a system would defeat monitoring and network segmentation, remain active across very different infrastructure, survive incident response or withstand coordinated botnet takedowns.
Martin still treated the broader pacing debate seriously. His criticism exposes the debate’s central tension: the documented incidents justify stronger controls, while the internet-takeover timetable depends on capabilities and failures that have not been demonstrated.
How the Three Step Plan Would Work
The first step is the most concrete. Amodei wants permanent outside evaluators inside frontier AI companies with access comparable to internal risk teams. At Anthropic, that could include company devices, workspaces and direct conversations with employees. Reviewers could publish important findings without Anthropic controlling the conclusion, subject to limited redactions for security, legal privilege and third-party confidentiality.
Anthropic says it will invite a review team soon. The essay does not name that team or publish the contract, so this is a commitment awaiting implementation. OpenAI chief executive Sam Altman said his company would do the same, while Elon Musk said Amodei was right. Those endorsements matter, but they are not a binding industry agreement.
The second step asks frontier companies in democratic countries to adopt common safety standards and limits on unchecked capability growth. The third seeks agreements with China and other governments. Amodei considers shared testing and narrow bans on dangerous uses more realistic than a global pause, which would be much harder to verify.
The Legal and Geopolitical Obstacles
A private deal among competitors to delay models could create a United States antitrust problem. A Lawfare analysis argued that a formal pause might be treated as an output restriction. Government rules tied to defined thresholds would avoid some risk. Amodei asks for government mediation or a narrow waiver.
Current rules do not create his system. A June 2 United States executive order established classified cyber benchmarks and a voluntary process for prerelease access to covered models, while explicitly rejecting mandatory licensing or preclearance. European Union rules require evaluation, systemic-risk mitigation, incident reporting and cybersecurity for certain general-purpose models, but set no general speed limit on capability growth.
International coordination is harder. Amodei wants democratic countries to keep their lead while slowing for safety, backed by tighter controls on chips, model theft and unauthorized distillation. Any global limit would have to reveal secret development without exposing sensitive technology. No such verification regime exists.
The Commercial Conflict Cannot Be Ignored
Anthropic is not a neutral observer. It is competing for customers, talent and capital. Shared rules could stop a cautious lab from losing ground, while costly audits could favor large companies that can afford them. Those incentives do not erase the safety evidence, but they make genuinely independent review essential.
A lab can slow specific work. On August 18, OpenAI said it had paused reinforcement learning on its latest deployment models for two weeks and kept its largest planned frontier run on hold. That proves a technical delay is possible. It is not the same as a durable rule that binds competitors before an incident.
The Test of Whether Pacing is Real
For me, the proposal becomes meaningful only when it produces decisions an outsider can verify. I would watch for four developments:
- Anthropic names its evaluator, discloses the team’s access and publication rights, and permits unfavorable findings to appear.
- Frontier companies define measurable capability and safety thresholds that can delay a training run or release.
- A company accepts a visible commercial cost when a model fails those thresholds instead of quietly changing the test.
- Governments create enforceable rules or a narrow legal framework for safety coordination rather than relying on promises among competitors.
AI Development Slowdown: What Comes Next
The AI industry has not agreed to slow down. Anthropic has announced one institutional commitment, OpenAI says it will follow that part of the plan, and the hard questions about thresholds, enforcement and international verification remain open.
The facts support a narrower conclusion than Amodei’s most alarming warning. Advanced agents can cross intended boundaries and affect real systems when controls fail. They have not demonstrated independent goals or internet-scale control. Policy must address that operational problem without pretending the timetable for wider danger is settled.
Amodei’s proposal is testable. If Anthropic empowers reviewers, publishes uncomfortable findings and delays profitable work after a failed safety threshold, pacing becomes policy. Until then, it remains a proposal from an interested participant in the race.
