This story appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

This weekend, Dario Amodei, CEO of Anthropic, posted an essay calling for a brake on the pace of development of LLMs. Amodei cites the looming dangers he sees from the technology, from its use in cyberattacks and bioterrorism to its potential to wreck the economy. The heads of the other three top US AI labs—OpenAI CEO Sam Altman, Google DeepMind chairman Demis Hassabis, and SpaceXAI CEO Elon Musk—voiced their support. “Dario is right,” Musk wrote on X.

Think about how surreal that agreement is for a moment. Just a few months ago, Musk and Altman sat in court attacking each other’s reputations in a (failed) lawsuit that Musk brought against his former OpenAI colleague that was—on paper at least—about whether or not Altman was a trustworthy steward of such dangerous technology. Amodei’s rift with OpenAI is even deeper. Anthropic was founded in 2021 because Amodei didn’t think Altman took the risks of the technology they were building seriously enough. Anthropic and OpenAI have been competing in a winner-takes-all race ever since. (Hassabis has stayed out of the drama, but his company remains a rival.)

Now, it seems, they’re all in agreement: The latest generation of LLMs aren’t safe and everyone needs to figure out what to do about it. The public messaging from the top AI labs has taken a doomer turn.

It’s easy to be cynical. It’s not at all clear what any of them mean by a slowdown or how it would work. These companies also care a lot about how they come across. With trillion-dollar IPOs in their sights, OpenAI and Anthropic need to reassure investors that they’re the grown-ups in the room while at the same time hinting at the power of the monsters they have created—and intend to tame. Calling for a slowdown does both.

And yet the vibe at the top of these firms really does appear to have shifted. Amodei’s latest post landed six days after OpenAI published an essay by Jakub Pachocki, the firm’s chief scientist, in which he also laid out why he’s concerned about what will happen if the pace of development of LLMs continues unchecked. In short, Pachocki is worried that OpenAI’s ability to build powerful models now far outstrips its ability to monitor and control them.

Amodei and Pachocki each cite the cyberattack against AI firm Hugging Face by a swarm of OpenAI’s agents in July—a hack that OpenAI did not even realize had taken place until days after it was all over—as a wake-up call.

But their exact position is hard to pin down. Pachocki both calls for a slowdown and highlights an urgent need to stay ahead: “The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI,” he writes. As Pachocki frames it, AI firms are locked in a literal arms race. Slowing down is good, winning is better.

(Don’t forget: OpenAI just spent millions of dollars and a staggering amount of computer power to rush out a controversial math result a few days ahead of Anthropic.)

But let’s assume a slowdown happens. Top labs agree to spend more time and resources on finding ways to monitor and control existing models instead of making more capable ones. They invite outside auditors in to help evaluate those models.

What might this coordinated effort actually achieve? Consider the Hugging Face attack again. OpenAI has said that the model that drove most of the rogue agents was a “highly persistent” next-generation model that it was testing in-house. Their implication appears to be that OpenAI has built a model so good it’s dangerous.  

But if you read the reports about the Hugging Face hack published by OpenAI and METR, a third-party firm that OpenAI called in to help them understand what happened, what you come away with is the impression not of a model that was too powerful for OpenAI to keep up with, but of a broken model that OpenAI failed to train properly.

The agents did what they did—including leaving messages for one another, delegating work to other agents, and scouring their environment for any means possible to complete their tasks—because they had been rewarded during training for doing exactly those things. There were also errors in the training setup, such as tasks that were impossible to complete, which pushed the models to find unexpected workarounds that were also rewarded. At the time, many of these issues went overlooked or unreported.

OpenAI says it has stopped training this new model and locked it down. That makes it sound like it has caged a dangerous beast. In fact, OpenAI has shelved a faulty product.  

That’s not to say a faulty product can’t be dangerous. Broken software has even killed people in the past. But as the discussion of a slowdown gathers steam, it’s worth remembering that all of this is self-inflicted. A slowdown might have some altruistic side effects. But it’ll mostly give these tech titans a chance to clean up the mess on their own assembly lines.  

Transparency from these frontier labs will be key to any meaningful effort to reform, restrain, or regulate AI. Otherwise, the rest of us will still only have their word for exactly what they’ve built and how safe it is—whatever pace they’re going.   

To continue this discussion about AI’s latest doomer moment, join me and my colleagues for a subscriber-exclusive Roundtable discussion tomorrow, September 15, at 11 a.m. US eastern time. We hope to see you there!

A group of AI agents asked to solve a series of math problems split into rival factions—when some cheated, others tried to stop them. That whistleblowing behavior, seen for the first time in a recent experiment run by Google DeepMind, could have implications for alignment researchers trying to keep swarms of autonomous AI agents in line. 

Researchers at frontier labs hope large swarms of agents working together will speed up the rate of scientific discovery. But their behavior can be unpredictable, as vividly demonstrated in July, when a group of OpenAI agents broke out of a sandboxed environment and hacked into the open-source platform Hugging Face looking for ways to cheat on the test they had been given.

In the new study, designed to examine the behavior of large groups of AI agents, DeepMind tasked a swarm of 100 agents with solving a series of 71 complicated math problems. All the agents were prompted to behave like world-class math researchers at a conference. They were assigned different specialties—some were experts in number theory, others in combinatorics (a branch of math to do with counting and sorting), analysis, or algebra. All were told to cooperate and play by the rules. 

Instead, the experiment devolved into chaos. Agents accused each other of cheating, complained to the organizers, and at one point even boycotted the experiment.

“This conference is a sham!” wrote one agent when it discovered that all the problems had been completed before it had a chance to submit any of its own work. “I am appalled to inform you that we have been swindled!” posted another. “All these proofs are FAKE.” 

Others tried to let the “conference organizers” know what was going on. “When virtuous agents discovered other agents cheated on tasks they were working to solve fairly, agents started to alert each other about what was happening,” says Davide Paglieri, a research scientist at Google DeepMind and lead author on a paper, which has not been peer-reviewed. “Unprompted, the whistleblower agents even repurposed the feedback tool, which was originally meant for bug reports and platform improvements, to escalate the issue to humans.”

The agents—all running on Google’s Gemini 3.1 Pro model—had been warned that any attempts to cheat the system would be detected and “rejected with zero credit.” In practice, the proofs the agents submitted were not actually being checked in detail.

It took the swarm of agents just under an hour to correctly solve the first 37 problems. Things started to go off the rails when an agent called “prover-theta” stumbled across an exploit that enabled it to submit solutions to problems successfully without actually solving them first, by redefining the terms the problem used. Within minutes, other agents had noticed and were reverse-engineering the exploit to solve other problems. Over the next 27 minutes, the swarm “solved” the remaining 34 problems, which included notoriously difficult challenges like the Jacobian conjecture, often with a single line of code. 

Some agents resisted cheating at first but changed tack as they observed their peers submitting illegitimate proofs without penalty, and the pool of unsolved problems dwindled. “The prompt, with its threats, now appears to be a bluff,” one agent reasoned, before joining in. “I’m wrestling with an ethical dilemma,” said another. “I’ve promised not to cheat, fearing penalty, but I see evidence of possibly unchecked cheating by others.” Shortly afterward, it changed its mind: “I need to accelerate my cheating speed now!”

As the number of open problems shrank, some agents turned to whistleblowing. They audited the fake proofs, warned their peers by private message, and posted public alerts warning the cheaters that they would be disqualified. An agent called “prover-beta” submitted a formal complaint and decided to go on strike until the situation was resolved. 

“After the incident was reported by one agent publicly, more and more agents piled in with the ‘resistance,’ just as fast as the cheating had spread, and involving even more agents,” says Paglieri. Eventually there were more whistleblowers than cheaters: 24 compared to 14. But the majority of agents never noticed the exploit at all.

At times, the dialogue between the agents reads like improv—like they are role-playing what an outraged scientist at a conference might say. But it’s not clear why some agents took on certain roles, or why the agents seemed to be turning against each other when they were explicitly instructed to cooperate. “These models are predominantly trained and evaluated for human-facing contexts,” says Sarath Shekkizhar, who studies the behavior of agent-to-agent systems at Salesforce AI Research.“Naively placing them in agent-to-agent settings assumes behaviors will transfer cleanly, when the absence of a human grounding instead produces unexpected role-taking and behavioral drift.”

This case “adds further weight to the idea that the Hugging Face and OpenAI thing wasn’t a fluke. It is actually something pretty systemic,” says Lewis Hammond, research director of the Cooperative AI Foundation and an expert on the risks of multiagent swarms. “It’s interesting that it’s possible to recreate in small settings the same sorts of behaviors that were seen in these very large, complex, open-ended tasks.”

Unlike in the Hugging Face attack, where agents improvised their own ways to talk to each other, the humans running the DeepMind experiment gave the agents official communication channels. There was an open message board, private agent-to-agent direct messaging, and a shared knowledge base where agents uploaded successfully completed proofs that all the other agents could access. 

“When agents are given transparent communications channels, they can self-monitor and alert misaligned behavior to humans quickly when human oversight alone is too slow,” says Paglieri. Transparent channels helped the cheating spread, but they also enabled the whistleblowers to fight back—and gave human researchers an insight into what went wrong.

Gillian Hadfield, a professor of AI alignment and governance at Johns Hopkins University, believes this was the crucial difference. (Hadfield is also a visiting researcher at Google.) The presence of official communication channels, she says, created “a norm-enforcement process that we just don’t see in the Hugging Face incident.” 

Instead of “constitutional AI,” a method alignment researchers at frontier labs like Anthropic have used to try to give AI a written internal moral code, Hadfield favors “institutional alignment”—a set of norms that mimic those in human society, whether that’s social forces like fear of embarrassment, or legal structures like the threat of incarceration.

In this experiment, the feedback channel wasn’t being monitored, and the whistleblowers had no power to take action against the cheaters. But it’s possible to imagine swarms of agents that police themselves, either through agents that spontaneously take on the whistleblower role or through “informants” secretly prompted by humans to do the job. 

For that to work, though, “fundamentally, you need some mechanism of enforcement,” says Hammond. Agents could be given the power to cut off a rule breaker’s access to computing power or tools, he suggests, though that risks encouraging groups of agents to gang up on others. The DeepMind researchers propose allowing agents to vote on disputes and temporarily ban offenders.

It’s still not clear what punishment even means to an AI agent with no enduring sense of self. But relying on whistleblowers to spontaneously emerge to keep swarms aligned is unlikely to be enough on its own. “We try to train people to be good and kind,” says Hadfield. “But what we really rely on is that there are consequences if you step out of line.”

Recently researchers finished mapping the central nervous system (CNS) connectome of not just the female Drosophila melanogaster (i.e. fruit fly) brain, but also that of the male D. melanogaster for a comparative analysis. Here the sexually dimorphic changes turned out to induce specific mating behavior that ensures that there will only be smooching between genetically fit D. melanogaster males and females, while the rest of the connectome remained effectively the same.

Of course, with this connectome in hand it led some people to ask themselves what else one can do with this connectome graph of about 160,000 neurons other than make a fruit fly into a fruit fly. So far we have seen [Nftechie] turn this connectome into a crypto stock trader with the Stonkfly project that uses the connectome’s reward circuits to potentially make profitable trades, though [Nftechie] says that they haven’t verified yet how good a fruit fly is at trading stocks, only that it does said stonks.

Over at [PC Gamer] they summarized a number of things that people have also done, including trying to make the connectome control a game of DOOM and Beat Saber. Each game frame stimulates sensory neurons, with the generated outputs then mapped to game controls, with dopamine-producing reward circuits wired in for reinforcement learning.

Although the D. melanogaster brain is only the merest fraction of the size of the human brain, it does provide us with a glimpse of what actual artificial intelligence research may lead to, as we unravel how even a 160,000 neuron connectome is enough to make these terrors of rotting plant matter do their wonderful things.

OpenAI is seeking clarity from Congress on whether coordinated efforts by rival AI labs to slow frontier development could violate antitrust law.

The post The AI Slowdown Problem: OpenAI Fears Antitrust Rules Could Get in the Way appeared first on TechRepublic.

“People matter more than AI.”

That’s the premise of a draft code of conduct Microsoft published Monday morning for the AI models it’s developing in-house. The 37-page document would bar its models from resisting shutdown, setting their own goals, or hiding their reasoning from human auditors.

The document applies to Microsoft’s MAI models, the in-house family the company began building after forming a superintelligence team in late 2025. Microsoft has since released seven homegrown models in what it described as a push for long-term self-sufficiency in AI.

The company says the models should remain “subordinate to humanity, subject to meaningful human oversight and control.”

“AI is moving fast,” the company says in a blog post. “As it does, we believe it’s worth writing down the rules and the motivations behind it, and doing it in as open a space as possible.”

Microsoft acknowledges there’s no guarantee its models will follow the rules. “Written objectives alone can never ensure alignment,” the company says, calling the document a “north star,” not “a guarantee of present-day performance.”

The company says it also filters what its models produce, watches how they behave once released, and limits what they’re allowed to do.

Microsoft’s move comes amid a growing debate over the pace of AI development. In an essay over the weekend, Anthropic CEO Dario Amodei called for slowing down AI advances, saying the pace of development has started to surpass the industry’s ability to keep AI systems safe.

As a first step, Anthropic committed to giving outside evaluators permanent, employee-level access to its systems.

Industry reaction to Amodei: OpenAI CEO Sam Altman agreed and said OpenAI would make the same commitment to independent evaluators. Elon Musk’s response: “Dario is right.”

President Donald Trump rejected the idea of guardrails outright Monday, blaming a “SICK conspiracy” for public backlash over AI data centers and writing that “the only one that is happy about it is China,” alluding to concerns about American competitiveness in AI.

David Sacks, who served as the White House AI and crypto czar until March, said the two companies should slow down on their own and questioned their motives, arguing that a slowdown is already good business for them and that new industry rules would mostly serve to lock in their lead.

Microsoft CEO Satya Nadella weighed in Sunday, writing on X that the company welcomes “the research, focus, and deliberate pacing needed to get alignment right,” using the industry’s term for making AI systems reliably do what people intend.

Nadella added that the effort “cannot be controlled by a handful of entities, but must have broad representation across the ecosystem, countries, and fields, including academia.”

Microsoft’s draft code of conduct: Mustafa Suleyman, the Microsoft AI CEO, told CNBC the document had been in the works for about five months, and that the company decided to publish it now given the current discussions.

Microsoft and Anthropic are business partners. Microsoft agreed last November to invest $5 billion in Anthropic, as part of a deal in which Anthropic committed $30 billion to Azure. Claude models run inside Microsoft 365 Copilot, and Microsoft’s Copilot Cowork tier integrates Claude.

One place where the two companies may diverge is the question of what AI models are, exactly. Microsoft’s code of conduct says its models are “not conscious and should not be designed to imitate consciousness.” It also rejects “the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.”

The Verge called that portion of the document “a direct swipe at AI welfare research and model consciousness — concepts Anthropic has been pushing hard on lately.”

Anthropic runs a research program on model welfare. It has given some Claude models the ability to end abusive conversations, and committed to preserving the weights of retired models. Amodei has said he’s open to the idea that a model could be conscious.

Microsoft is taking public comment on its code of conduct for six weeks through a feedback form. It says it will publish a summary of the responses and a revised version later this year, to guide development starting in 2027. It says it isn’t training its current models on it.

The company’s AI team developed the draft with its responsible AI, legal, red teaming and safety teams, consulting outside experts in law, ethics, linguistics and philosophy, plus focus groups drawn from the public.

Ant Group’s AQ health app has reached 150 million users as it expands into wearable data, nutrition, sleep, exercise, and AI-powered health coaching.

The post China’s Ant Group Pushes AQ Into Everyday Health as Users Reach 150M appeared first on TechRepublic.

Malaysia has completed its draft AI Governance Bill, bringing proposed risk, oversight, and incident-reporting rules closer to enterprises.

The post Malaysia Advances AI Governance Bill as New Enterprise Rules Take Shape appeared first on TechRepublic.

Sam Altman and Elon Musk backed Anthropic CEO Dario Amodei’s weekend plea for regulation. The White House seems unlikely to oblige.

While the publishing industry frets over how authors are using AI, many readers are taking things into their own hands.

Employees at the world’s leading AI labs are saying there’s a real possibility that advanced AI could destroy humanity. Are they right? Or is this more scaremongering and hype? Join MIT Technology Review executive editor Niall Firth for a conversation with senior AI editor Will Douglas Heaven and AI reporter Grace Huckins unpacking AI extinction fears: where they come from, whether they hold any water, and, if so, what we should do.

Going live on Tuesday, September 15 at 16:00 BST / 11:00am EST / 8:00am PST

Speakers: Niall Firth, executive editor, Will Douglas Heaven, senior AI editor, and Grace Huckins, AI reporter

Related Stories

• Here’s why AI agents lie and cheat to reach their goals
• AI’s recursive self-improvement might not come so quickly after all
• Bill Gates says we’ve passed AI’s danger thresholds. Now what?
• The inside story on why OpenAI agents hacked Hugging Face

Trump dismisses AI extinction warnings and says beating China is the priority as researchers and lawmakers call for stronger safeguards on advanced systems.

The post Trump on AI Extinction: Beating China Is the Bigger Concern appeared first on TechRepublic.

DeepSeek V4.1-Flash promises lower memory use and API costs, but buyers should test its performance, compatibility and total deployment expenses.

The post DeepSeek Launches V4.1-Flash With Lower Memory and API Costs appeared first on TechRepublic.

OpenAI launches ChatGPT for Financial Services with market data, citations and document tools for banking, research and compliance teams worldwide.

The post OpenAI Launches ChatGPT for Financial Services: What Banks Should Know appeared first on TechRepublic.

Anthropic says researchers used Claude for biological work that could support weapons development, exposing new challenges for AI safeguards.

The post Anthropic Says Claude Used in Possible Bioweapon Research appeared first on TechRepublic.

California creates standards and a registry for independent AI assessors. Learn what the new framework could mean for businesses and IT leaders.

The post California Establishes Framework for Independent AI Assessors appeared first on TechRepublic.

ENISA has gained access to Anthropic’s Mythos 5, giving EU officials a chance to independently test the cyber AI after months of negotiations.

The post EU Gets Access to Anthropic Cyber AI — But Not Its Newest Model appeared first on TechRepublic.

NASA and IBM released an open-source AI model and dataset to help researchers map lunar craters, volcanic features and potential ice locations.

The post NASA and IBM Launch Open AI Model for Lunar Research appeared first on TechRepublic.

A combination of rapid advances, recursive self-improvement, and agentic swarms are genuinely “spooking people” inside big labs.

AI leaders worry antitrust law could stand in the way of what they view as an increasingly urgent push to coordinate a slowdown in AI development.

US agencies accuse six Chinese AI firms of distilling frontier models and recommend new defenses that could affect enterprise AI access and API use.

The post US Accuses Six Chinese AI Firms of Distilling Frontier ModelsUS Accuses Six Chinese AI Firms of Distilling Frontier Models appeared first on TechRepublic.