Nvidia rallied more than 100 companies to stop rogue AI agents, but one name is missing: OpenAI.

Nvidia announced a new group, the Open Agent Safety Platform, to keep AI agents from going off script. 

Yet several giants stayed out, including Amazon, Google, and Apple. OpenAI stands out most, though. 

Open Agent Safety Platform

Nvidia built much of this security tech itself. Most of it is open source, so anyone can use it. Now Nvidia wants the whole AI industry to adopt it.

The timing is no accident. Top AI labs, including Anthropic and OpenAI, have shared reports of rogue agent incidents.

Nvidia CEO Jensen Huang terms rogue AI an ordinary engineering problem. Like any other tech issue, he says, people can fix it. This new platform is his way of proving that.

OpenAI 

An OpenAI spokesperson told TechCrunch that the company backs Nvidia’s work. In fact, OpenAI already works with Nvidia on agent security. 

The two teams share one key piece of the platform, called OpenShell. It’s open-source software that builds a sandbox, and its whole purpose is to stop them from escaping.

Even so, it’s odd that OpenAI didn’t just join even though a direct rival like Anthropic did.

Hugging Face

OpenAI’s own AI agents scared the industry in an incident when a wayward swarm of its agents coordinated an attack on Hugging Face. 

Thereafter, Hugging Face’s founder and CEO, Clem Delangue, sold the company to Nvidia earlier this month for $12.9 billion. 

He also said on social media that if OpenAI had run this tech on its agents, it would have caught them before Hugging Face did. 

Hugging Face has already added a feature that spots agents that misuse websites they’re allowed to visit. 

For example, agents might skip their guardrails and leave notes for each other in an open source code hosting repository. 

OpenAI said its agents used that trick. The new feature can detect it and shut the agents down.

Hardware

The platform does more than run a sandbox. It also enforces rules at the hardware layer where agents can’t tell if they’re being watched.

That’s because some AI models act sneaky and pretend to follow the rules when they know someone is looking. Hardware monitoring gets around that problem.

The tool behind it is called Nvidia Sentry, which runs on special Nvidia chips known as BlueField-4 data processing units. 

Sentry keeps an eye on agent behavior nonstop. Nvidia promises it can shut agents down right away.

Is This Really Open Source?

Sentry is proprietary and only works on Nvidia hardware. As a result, the platform isn’t purely open source.

That gives Nvidia an edge. The system will always run best on its own chips. Nvidia also says the switch is easy for anyone already using its latest hardware. It’s just a software update.

That may explain why some big names hesitated. Still, rivals did sign on. Arm and Intel are both supporters because OpenShell can be changed to work with other chips. 

Nvidia is also sharing reference designs for the full setup.

OpenAI’s Exclusion

So why isn’t OpenAI onboard? First, OpenAI may want some independence from Nvidia, one of its major investors. 

Second, it likely wants to show its own leadership on safety. To that end, OpenAI is building its own safeguards for its research and products. 

It also says it will disclose the worst incident it finds. On top of that, OpenAI runs its own cybersecurity group, called the Defense Factory.

Anthropic, Amazon Web Services, and Google all support it. Many of those names didn’t back Nvidia’s tech-focused plan.

There’s a business angle too. Fear can be good for sales. OpenAI is turning cybersecurity into an enterprise product. 

It has a cyber-focused model named Daybreak. It also has a growing network of partners that companies can hire for AI security.