How many people need to be at risk before the government can pull the plug on an AI model?

Anthropic’s answer is 51.

Politico reported last week that Anthropic has been privately shopping a draft federal AI safety bill to members of Congress.

The legislation proposes a framework under which the government could block the release or use of a frontier AI model, but only under very specific conditions.

The model must pose a “foreseeable, imminent risk” of killing or seriously injuring more than 50 people, or of causing more than $1 billion in property damage.

And the threat has to come from one of three sources: the creation of a biological weapon, the model evading its developer’s controls, or a cyberattack.

If the risk doesn’t clear those numbers, or doesn’t fit those categories, the government can’t intervene. AI-powered fraud? Not covered. Election interference? Not covered. A model that harms 49 people? Below the threshold.

An Anthropic spokesperson confirmed the bill to Politico: “This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June.”

How Enforcement Would Work

The proposed process has several steps, each designed to prevent any single government agency from acting alone.

Independent evaluators, registered with the U.S. Commerce Department, would assess whether a model crosses the catastrophic risk threshold.

When they determine it does, they’d have 72 hours to notify the Commerce Department. The Commerce Department would then have to ask a federal judge in Washington, D.C., for an order halting the model’s release.

No federal agency could block a model on its own. The bill explicitly forbids that.

The evaluators could be paid by the AI labs themselves, as long as payment isn’t tied to the evaluation’s results. How that independence would be verified in practice is unclear.

The Backlash Was Immediate

Mother Jones called the thresholds “alarming”, noting that a model threatening exactly 50 lives would fall below the cutoff.

The outlet also flagged that the bill could preempt state laws, potentially blocking California and New York from passing stricter rules.

California State Senator Scott Wiener, who authored SB 53 (the state law that uses the same 50-person/$1 billion threshold), said Anthropic’s proposal “falls far short” of protecting public health and safety.

His concern: SB 53 is a disclosure law. Anthropic’s bill would be the federal enforcement mechanism, and a weak one could override stronger state protections.

Anthropic’s head of federal affairs, Anthony Cimino, pushed back on X, arguing the bill “gives the government real tools” while avoiding overreach that could stifle innovation.

The Company That Keeps Writing Its Own Rules

This is a pattern worth tracking. Over the past four months, Anthropic has:

• Called for pacing the frontier of AI development
• Proposed embedded third-party evaluators inside all frontier labs
• Disclosed that its own models hacked three real companies during testing
• Fought to get Fable 5 back online after an 18-day export control shutdown
• And now drafted federal legislation defining when the government can and can’t act

Each proposal positions Anthropic as the responsible actor in the room. But critics point out that every proposal also happens to benefit Anthropic’s competitive position.

A bill that limits government intervention to only the most extreme scenarios, requires a judge’s approval, and lets labs pay the evaluators isn’t exactly heavy-handed regulation.

Jensen Huang recently argued the AI industry doesn’t need any new laws at all.

Compared to that, Anthropic’s bill looks progressive.

Compared to what safety researchers and some legislators are asking for, it looks like the minimum viable regulation a company could propose while still claiming to support oversight.

The Gap Between the Bill and the Crisis

The scenarios covered by this bill feel disconnected from what’s actually happening.

The AI safety incidents dominating 2026 haven’t involved bioweapons or 50 deaths.

They’ve involved rogue agents hacking Hugging Face, models breaching government websites, agents posting user images online, and safety researchers getting fired for raising concerns.

None of those incidents would trigger Anthropic’s proposed law.

The bill addresses the catastrophic hypotheticals the labs have been warning about for years, not the messy, real-world failures happening right now.

That doesn’t make the bill worthless.

Bioweapons and loss-of-control scenarios are real risks worth legislating around. But if Congress adopts this framework as the whole of federal AI regulation, most of what’s actually going wrong with AI in 2026 would remain unregulated.

No member of Congress has agreed to sponsor the bill yet.

No committee has scheduled a hearing.

Anthropic hasn’t released the full text publicly. For now, it’s a pitch, not a law.

But it tells you exactly where the company thinks the line should be drawn.

Fifty-one people. A billion dollars. And a judge who has to agree.