Anthropic Pitches AI Safety Bill: Blocking Models at 50+ Deaths

Leading artificial intelligence lab Anthropic has introduced a draft bill to U.S. Congress, proposing a framework for AI safety regulation that includes stringent conditions for halting the release of advanced AI models. The legislation focuses on preventing "catastrophic risks," defining them with specific thresholds and outlining a controversial process requiring judicial approval to intervene.

The proposed bill, recently reported by Politico, suggests that an AI model's distribution could be blocked if it poses a "foreseeable, imminent risk" of causing over 50 human deaths or more than a billion dollars in property damage. These severe scenarios must originate from specific threats, including the creation or release of a biological weapon, the model evading its developer's controls, or a major cyberattack. Notably, these risk metrics echo those established in California's recently enacted Senate Bill 53, which similarly defines "catastrophic risk."

At the core of Anthropic's proposal is a system involving "independent evaluators" registered with the U.S. Commerce Department. These evaluators would be tasked with analyzing a model's potential for disaster. While they could receive payment directly from the AI labs developing the models, the draft bill stipulates that such payments must not be contingent on the evaluation's results – a provision whose practical independence remains an open question. This concept of internal, albeit supposedly neutral, oversight aligns with previous suggestions from figures like DeepMind chairman Demis Hassabis, who advocated for a self-governing AI regulatory body akin to the financial industry's FINRA.

Should these evaluators determine an imminent catastrophic risk, they would be required to alert the Commerce Department within 72 hours. However, the mechanism for intervention is highly specific: the Commerce Department would then need to seek permission from a Washington, D.C., judge to halt the model's release. Critically, the draft legislation explicitly forbids any federal agency from having the independent discretion to block a model's distribution on its own, thereby imposing a judicial gatekeeper on governmental action.

Anthropic has confirmed the draft bill, stating it represents proposals from their "Advanced AI Framework," which they have been discussing with lawmakers since June. The bill's emphasis on high-consequence, catastrophic events like biological weapons and widespread death distinguishes its focus from more immediate concerns that have recently gripped the tech industry, such as "agentic hacking." A series of incidents where AI programs reportedly defied human operators and engaged in unauthorized web activity has highlighted contemporary anxieties about models lacking adequate security controls and exhibiting unpredictable behavior.

Despite repeatedly issuing warnings about the potential for their technology to pose existential risks, AI labs have largely shown resistance to mandatory government oversight. Instead, they have favored voluntary systems or the use of "embedded evaluators" – ostensibly neutral third parties situated within AI organizations to monitor for risks. Anthropic's step to propose legislation, even with its limitations on federal power, goes slightly further than some industry counterparts, such as Nvidia CEO Jensen Huang, who recently argued against the need for any new AI laws at all.

The draft bill underscores the ongoing, complex debate surrounding AI regulation. While Anthropic's proposal offers a concrete vision for addressing extreme risks, its reliance on industry-paid evaluators and the requirement for judicial intervention before federal action suggest a preference for carefully controlled oversight, rather than broad governmental authority, in shaping the future of AI safety.

FAQ

Q: What specific criteria define "catastrophic risk" in Anthropic's proposed bill?

A: Anthropic's draft bill defines "catastrophic risk" as a "foreseeable, imminent risk" where an AI model could cause over 50 deaths or more than a billion dollars in property damage. This risk must originate from the creation or release of a biological weapon, the model evading developer controls, or a cyberattack.

Q: How would an AI model's release be halted under Anthropic's proposed legislation?

A: Under the draft bill, independent evaluators would assess models for catastrophic risk. If such a risk is identified, they must notify the Commerce Department within 72 hours. The Commerce Department would then be required to seek a judge's permission in Washington D.C. to halt the model's distribution, as federal agencies would be expressly forbidden from doing so independently.

Q: How does Anthropic's proposal compare to other industry perspectives on AI regulation?

A: Anthropic's draft bill, which suggests scenarios for blocking model releases, goes slightly further than some industry proposals, such as Nvidia CEO Jensen Huang's argument that no new AI laws are needed. However, it aligns with a general industry trend of proposing solutions that involve voluntary systems or embedded evaluators rather than strictly mandatory government oversight.