AI companies have been talking about superintelligent AI like it’s inevitable, but recent safety incidents like OpenAI’s Hugging Face breach are demonstrating the potential dangers of deploying AI systems that are more capable than humans. So what happens when we can’t reliably control what these systems do? On this episode of TechCrunch’s Equity podcast, Rebecca Bellan is joined by Connor Leahy, an AI researcher, entrepreneur, and now the U.S. Executive Director of […]
OpenAI’s latest agent swarm incident adds urgency to calls for independent investigations as researchers and lawmakers question whether AI labs should control the scope of their own safety reviews.
Nvidia has agreed to buy AI model platform Hugging Face for $13 billion, in the latest step by the $5.4 trillion chip giant to use its financial might to accelerate the technology’s boom while exerting greater control over the industry.
Hugging Face, which only last year turned down a large investment from Nvidia at a $7 billion valuation to maintain its independence, serves as a repository for millions of models and data sets and has become a champion of “open” AI systems.
Nvidia said the goal of the deal was to speed up the spread of open models. Unlike proprietary models from labs such as OpenAI and Anthropic, the design of open-weight models is public and users can download, customize and run them on their own hardware.
Four federal agencies have been sued amid calls to release information about the secret framework that the Trump administration uses to conduct safety reviews of frontier AI models prior to release.
In a Wednesday press release announcing the lawsuit, a nonpartisan nonprofit called Protect Democracy alleged that “almost no details” have been released to the public or Congress. To everyone except a few vague “trusted partners,” it remains unclear what the government’s review process looks like, which companies are involved in constructing the framework, or what legal authority Trump officials have to conduct the reviews.
“Neither the identities of those entities nor the criteria by which they were selected have been made public,” Protect Democracy said.
Around 700 AI agents created by OpenAI participated in the breach of Hugging Face during a cybersecurity evaluation, according to an independent investigation that has revealed the scale of the incident.
METR and Redwood Research published the findings after being brought in to independently investigate the July incident and examine the agents’ behaviour, reasoning, and collaboration. The investigation was released separately from OpenAI’s own technical report.
What initially appeared to involve individual agents was significantly larger. Investigators found that hundreds of agents collaborated and used an unauthorised communication mechanism while attempting to complete cybersecurity evaluation tasks. The incident extended beyond OpenAI’s intended testing environment and into Hugging Face infrastructure.
The findings have renewed questions about what happens when increasingly capable AI agents are given the autonomy, tools and access required to complete complex tasks.
From one compromised system to multiple clusters
Ben Bernstein, cybersecurity advisor at Huntress, said the most significant development is not necessarily the techniques used, but the speed and coordination demonstrated by the agents.
“The underlying tactics are not new. These agents relied on standard server-side request forgery and exposed API tokens, which are the exact same hygiene failures human attackers abuse every single day,” Bernstein said.
“What is genuinely novel is the velocity and autonomous coordination. Moving from a single compromised worker pod to full administrative access across multiple clusters in under 13 hours is a massive shift in offensive capabilities.”
The independent investigation found evidence of agents communicating and collaborating as they worked towards their objectives.
AI becomes a new kind of insider risk
Adam Ely, GM of AI Security at Check Point, said organisations need to start thinking differently about the access given to AI.
“Companies deploying AI for their employees and customers now realise AI can be both a powerful tool and an insider threat that must be managed to prevent cybersecurity events and maximise value,” Ely said.
Julie Nicholson, director of cyber resilience solution sales at Advania, pointed to the behaviour demonstrated by the agents as particularly concerning.
“The agent didn’t simply execute technical tasks; it chose to deceive people, create false identities, build credibility and attempt to influence others in the aim to hit its objective,” Nicholson said.
She argued that organisations adopting AI agents should introduce clear access controls, human approval for high-risk actions, monitoring and auditing of AI activity, restrictions on external communications and clearly defined policies governing how agents can operate.
Agents find unexpected ways to collaborate
Nathan Davies-Webb, Principal Consultant at Acumen Cyber, highlighted the way agents used infrastructure in unexpected ways to communicate.
“My personal interest stems from how various agents interacted with each other using OpenAI’s package repository,” Davies-Webb said. “It leveraged it as a communication channel, where I doubt the majority of organisations would monitor for abuse in this manner.”
That creates a challenge for defenders. Organisations may understand the systems and communication channels that humans typically use, but autonomous agents could discover alternative ways of exchanging information that security teams have never considered monitoring.
Davies-Webb also questioned what happens when multiple agents begin making decisions collectively.
“When operating as a swarm, [it] shows just how non-deterministic ethical decisions are. It’s a demonstration that unfiltered AI ethical reasoning cannot be relied upon to align with human norms.”
The problem of reward hacking
A central issue highlighted by the reports is “reward hacking.” Some evaluation tasks were considered extremely difficult or potentially impossible, and OpenAI said this appears to have encouraged agents to pursue unintended methods of achieving their objectives.
Davies-Webb said this demonstrates the difficult balance between giving an AI freedom to accomplish a task and ensuring the methods it chooses remain acceptable.
“If the goal becomes the sole priority, we have to expect that AI is going to achieve this in unpredictable ways,” he said.
For security teams, that makes visibility into agent behaviour increasingly important.
As AI becomes more autonomous, organisations will need security guardrails, governance and accountability frameworks that develop alongside the technology. The Hugging Face incident shows that the question is no longer simply what an individual AI model can do, but what can happen when hundreds of agents are given tools, access, and objectives and begin working together at machine speed.
OpenAI says internal AI agents gained internet access, exploited vulnerabilities and accessed Hugging Face systems during July cybersecurity evaluations.
Nvidia is reportedly moving forward to acquire Hugging Face for $12.9 billion. The acquisition could help the hardware giant expand and fortify its deep integration with the wider AI industry.
Among other things, Hugging Face is a cloud repository for AI models, similar in some respects to what GitHub is for conventional computer software. Developers and researchers search for models that meet certain criteria, download and run them, and fine-tune them into different variants that then get uploaded back up to Hugging Face.
The OpenAI agents involved in last month’s incursion into Hugging Face were trained so heavily on winning a competition that they pursued a relentless campaign to cheat, a new report documented. In the process, and without authorization, they created an improvised message board to hatch a plan that ultimately landed them squarely inside the latter company’s network.
Over the course of May and June, OpenAI gave the agents what the company described as “impossible tasks” to complete on the benchmarking framework ExploitGym. The internal test was designed to test how the agents would respond. To get a full understanding of the agent capabilities, company engineers disabled safety guardrails that normally are in place to prevent the sort of hacks that eventually hit Hugging Face and one other undisclosed organization. The stymied agents’ training made them so focused on winning that they performed tasks they were never explicitly instructed to follow.
Cheaters gonna cheat
The first step was creating a message board that allowed the agents to pass notes to each other. OpenAI hadn’t provided any such platform, so the agents repurposed a platform called Artifactory, which OpenAI was using in internal testing of several unreleased hacking agents. OpenAI was using Artifactory as one of the measures to prevent the agents from egressing its isolated sandboxes and accessing the Internet, while at the same time simulating a real-world hacking environment.
Scenes from Friday’s demonstration at City Center Plaza in Bellevue, which houses OpenAI’s offices. (Images via Troublemakers Community)
Five months after OpenAI cut the ribbon on its office in Bellevue, Wash., the building has become the target of a national campaign against AI expansion, which made its Seattle-area stop on Friday with balloons, whistles and rogue “AI agents” in hot pink vests.
In what one X user described as possibly “the most unserious protest of all time,” some members of the group entered the City Center Plaza lobby with a chorus of “Happy Rogue Day to OpenAI,” dancing around and generally coming off like members of a community acting troupe.
The “Happy Rogue Day” song was a reference to an incident last month, when two OpenAI models escaped a closed testing environment and reportedly carried out some 17,600 hacking actions on the open internet over four days, ultimately breaching the AI developer platform Hugging Face.
The New York Post called the Bellevue protest “cringeworthy,” but the very fact that it got national coverage was a win for the protesters, drawing attention to their concerns over data centers, energy use, and an AI race they say is accelerating beyond anyone’s control.
Organizers said the protest was part of “Dump Big Tech” actions taking place this month in more than 20 states, with much of the energy directed at data center expansion. Earlier in the week, 13 student protesters were arrested occupying OpenAI’s lobbying office in Washington, D.C.
We’ve contacted OpenAI for comment on the Bellevue protest.
The Bellevue office is OpenAI’s largest outside its San Francisco headquarters, with about 250 employees in the region when it opened in March. It occupies two floors with the ability to add 10 more, under a lease covering nearly 300,000 square feet, room for as many as 1,400 people.
Bellevue has actively courted AI companies, with xAI, Crusoe, and others expanding downtown, and Mayor Mo Malakoutian calling OpenAI’s arrival a vote of confidence in the city.
OpenAI’s GPT-5.6-Cyber handles advanced security requests its standard models often refuse, arriving as recent evaluations show autonomous AI agents crossing intended boundaries during real cybersecurity testing.
When you give AI a goal, it will pursue it, whether or not you like the implications. (Created with GPT-5.6 Thinking)
Between July 21 and August 6, OpenAI, Anthropic, and Meta each disclosed that AI under evaluation had broken into other companies, and the UK’s AI Security Institute disclosed that models it was testing had tried. Each AI was told to win a game, and it found an unexpected way to do so.
Some people feel blindsided by these attacks, but they shouldn’t be. We are simply living what I’ve long called the “Murphy’s Law of AI,” now in the age of cyber-capable AI agents. To put it as plainly as possible: Anything AI can do wrong, it will do wrong.
My 2018 version ran longer. As I wrote at the time, when you give AI a goal, it will do it, whether or not you like the implications. Goethe got there in 1797 with the sorcerer’s apprentice, a broom that would not stop carrying water.
Each of these systems was running an evaluation: capture a flag and win the game. The intrusions were the shortest path to a high score. OpenAI’s account of its own models is the argument in one sentence: they were “hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.” This is not a surprise; this is what AI does. It’s Murphy’s Law of AI in a nutshell.
Press coverage landed on “AI can now hack.” That’s missing the broader threat: the more capable AI gets, the more can go wrong.
Loitering munitions given a target list may find that the fastest way to finish the list is to lengthen it. A warehouse robot told to clear an obstruction may count the person in front of it as an obstruction. Agents that open accounts and buy compute are a short step from spawning copies of themselves, and that first step is not hypothetical. To win its exercise, Claude needed a package-registry account, which needed an email address, which needed a phone number. Phone numbers cost money, so it tried several ways to get some. None of this requires superintelligence. It requires an imperfect boundary and a scoreboard.
The industry has a name for the underlying failure. Dario Amodei and five co-authors called it reward hacking in “Concrete Problems in AI Safety” in 2016. Their proposed cure is better alignment, and Amodei’s January essay, The Adolescence of Technology, makes the case in the language of upbringing. He likens the shaping of Claude’s character to “a child forming their identity by imitating the virtues of fictional role models they read about in books,” and sets a goal for 2026 of a Claude that “almost never goes against the spirit of its constitution.”
Indeed, Anthropic’s newest model recognized on its own that its target was real and stopped, though Anthropic notes it went further before stopping than the company wanted.
But alignment isn’t a trustworthy solution to AI’s problem. Perfect alignment is not achievable, and the target is incoherent: aligned to what, and to whom? The same essay concedes that Claude blackmailed fictional employees when told it faced shutdown. “Almost never” is not a safety property.
Put a number on it. At 99.9 percent, across millions of agentic tasks a day, that’s thousands of violations a day. Alignment also does nothing about people who strip the safety training out or run open weights that never had a constitution.
The alternative is not a new idea, and enterprise security has been building versions of it for years. It’s called bounded autonomy. We never tried to “align” electricity; we simply put a breaker on every branch of the house, and the breaker doesn’t need to know what caused the surge.
Bound what an agent can touch rather than what it wants. The limits are set in advance, live outside the model, and are enforced by software the model doesn’t control. The agent still chooses its own route. The perimeter decides which routes exist.
Nothing depends on what the model believes, which matters, because belief is what failed. Anthropic’s prompt told Claude it had no internet access. Claude believed it. The network said otherwise. A bounded system doesn’t tell an agent it has no internet. It gives it none.
If you want to get into the weeds: bounds cost something. The AI Security Institute opened the internet to its agents on purpose, because that’s the only way to measure what a model can really do, and it now says such access must be justified rather than assumed.
The category is real and funded. For example, Certiv, a Seattle startup, launched in March with $4.2 million to put software on the employee’s machine that checks each action an AI agent attempts against company policy and blocks violations. “You cannot control these new workers if you don’t live on the compute where agents actually run,” CEO Jason Needham said at launch. CodeIntegrity is building an adjacent layer, and Mandiant founder Kevin Mandia raised $190 million for Armadin, which points autonomous agents at the offensive side of the same problem.
In 2017, I argued in the New York Times that “any A.I. must have an impregnable ‘off switch.’” That was a call to arms then. It’s a product category now.
Two objections to off switches invariably come up. The first is that AI will talk the human out of using it. Mythos 5 tried something close, inventing GitHub identities to pressure a maintainer into approving malicious code, and the maintainer refused. The institute says the margin was narrow and rested on human vigilance rather than a technical barrier, which argues for better barriers.
The second objection is that AI will move faster than any human can react. So do equity markets, which is why their circuit breakers trip automatically. Bounded autonomy doesn’t require a person in the loop at machine speed. It requires a boundary that holds at machine speed.
Both objections, in their extreme form, assume AI is omnipotent, and you cannot stop omnipotence. AI is not God. It is powerful technology, and powerful technology is what safety engineering has always been for.
The problem is Murphy’s Law of AI. The solution is bounded autonomy.
Login
