OpenAI’s latest mathematical milestone has quickly become mired in controversy. Today, the company announced that its agents have solved one of the Millennium Prize Problems, some of the most important open problems in mathematics. Under normal circumstances, that solution would be a huge feather in OpenAI’s cap.

But the announcement has been overshadowed by accusations that OpenAI used NYU mathematician Tristan Buckmaster’s and Anthropic employee Levent Alpöge’s AI-assisted work on the problem as a jumping-off point and failed to credit them. OpenAI has denied the accusations.

It remains uncertain if OpenAI’s models made use of the work completed by Buckmaster and Alpöge, though Sébastien Bubeck, a member of the technical staff at OpenAI, said in a press briefing that the team was inspired to pursue the problem after hearing a rumor about Buckmaster and Alpöge’s efforts. But whether or not OpenAI’s models took advantage of Buckmaster and Alpöge’s research, this episode may mark a turning point in the history of mathematics.

AI models now seem essential for making progress on the most important mathematical problems of our time, and solving them may demand resources only available at a couple of frontier AI companies, which often defy the norms of academic collaboration that undergird most mathematical progress. If that’s the future we are headed for, it is unclear how human mathematicians will fit into it. 

The problem that OpenAI claims to have solved is known as the Navier–Stokes existence and smoothness problem. It is one of seven Millennium Prize Problems selected by the Clay Mathematics Institute in 2000. Solutions come with a one million dollar prize; before today, only one other Millennium Prize Problem had been solved. 

The Navier–Stokes problem concerns a set of equations that describes how fluids, such as water and air, flow over time. The equations are widely used in the field of fluid dynamics, and they have proven powerful, but physicists and mathematicians didn’t understand them completely. In particular, it was unknown until today whether the equations might, under some conditions, break down and predict an impossible state of affairs—such as a fluid having infinite velocity.

On Monday, NYU’s Buckmaster posted a proof on the social media site Mastodon showing that a simplified version of the Navier–Stokes equations can indeed break down—a major step forward on the Millennium Problem. He and Alpöge had worked on the problem for almost a year, using publicly available models from both OpenAI and Anthropic.

Then today, OpenAI presented a proof showing that the full Navier–Stokes equations can break down as well. The proof was obtained using an internal model that dramatically outperforms the already-impressive Astra model, which was only released last week. The company says it does not plan to claim the million-dollar prize for solving the problem.

These mathematical achievements are indisputably impressive, but they have attracted far less attention than the controversy about their origins. Along with the proof, Buckmaster posted a document detailing his interactions with OpenAI employees after he heard rumors about their work and reached out to one of them. According to him, OpenAI employees presented two possibilities to him: Either he and Alpöge could post their work and OpenAI would post their Navier-Stokes solution the following day, or he could work with OpenAI on a Navier-Stokes paper that excluded Alpöge from authorship, due to his affiliation with Anthropic, OpenAI’s biggest rival.

Buckmaster also wrote that he asked the employees whether the agents had obtained access to transcripts of the work that he and Alpöge had done with OpenAI models, which they denied; and whether OpenAI models had been trained on those transcripts, to which they offered no response. MIT Technology Review reached out to Buckmaster for comment, but didn’t hear back before publication.

The clear implication of the document is that OpenAI’s models somehow made use of Buckmaster and Alpöge’s work. That scenario is plausible on its face. The Buckmaster/Alpöge and OpenAI proofs both make use of an approach to the Navier-Stokes problem pioneered by the mathematicians Diego Córdoba and Luis Martínez-Zoroa.

According to Javier Gómez-Serrano, a mathematics professor at Brown University, this approach was one of several that was thought to hold promise for solving the Navier-Stokes problem. So, while it’s by no means impossible that both teams could have arrived at this approach independently, it’s also conceivable that Buckmaster and Alpöge’s work could have influenced OpenAI’s.

In the press briefing, Mark Chen, OpenAI’s chief research officer, again denied that any agents or OpenAI employees accessed Buckmaster and Alpöge’s transcripts—but given what has been revealed about the Hugging Face hack, it’s clear that OpenAI is not always entirely aware of what its agents are doing. 

If OpenAI’s models did train on Buckmaster and Alpöge’s work, or if its agents somehow gained access to it, then the company’s failure to track down the truth and assign those researchers appropriate credit reflects poorly on it. But there might be a thin silver lining to that version of the story for mathematicians, because it would suggest that the hard work of two humans, one of whom is a prominent expert on Navier-Stokes, was essential to the agents’ ability to solve the Millennium Problem.

Experts have long identified “research taste,” or the ability to choose promising research questions and directions, as a major obstacle for AI in science and mathematics. If the OpenAI agents did indeed choose to follow the Córdoba–Martínez-Zoroa approach because Buckmaster and Alpöge had done the same, then human research taste played an essential role in OpenAI’s success.

Even so, the bigger picture here is sobering. The progress that Buckmaster and Alpöge made over almost a year of collaboration with publicly available models speaks to the promise of human–AI collaboration. But they were not able to achieve a full solution. Meanwhile, OpenAI brute-forced a solution in a few days using an internal model, and their successful solution came at an astronomical cost: In the press briefing, Bubeck and Chen said the team was only able to solve the problem by running about 10,000 agents concurrently, at a cost of millions of dollars.

Over the past few months, I’ve heard from several researchers that mathematicians are becoming depressed, and it’s not difficult to see why. Mathematics is quickly becoming the province of frontier AI companies with impressive internal-only models, money to burn, and a lack of collaborative spirit. “Whether AI companies will decide to spend their money on doing one thing or another, I truly don’t know,” says Gómez-Serrano. “What is clear is that very few mathematicians will have resources of that scale.”

If OpenAI and Anthropic keep striving for more and more impressive mathematical accolades, there might not be any open problems left for human mathematicians outside of those companies to wrestle with. That would dramatically change the field of mathematics.

Last week, UCLA mathematician Terence Tao wrote a Mastodon thread describing how important mistakes, wrong directions, and incomplete solutions are for the field. “In most cases in pure mathematics, the problems are posed not because we desperately want the solution to these problems in and of themselves, but because we have seen from past experience that human-directed efforts to solve these problems tend to spur further development of the field,” Tao wrote.

“Prematurely solving the problem by purely AI-powered methods—particularly without full transparency into the solution process—can contaminate this process to the point where it actually becomes a net negative for the progress of mathematics as a whole.”

Humans might take longer than agents to solve mathematical problems, but in the process, they uncover new mathematical approaches and ideas that might inspire their peers and even birth their own subfields.

But when AI agents solve those problems instead—and when private companies keep the agents’ wrong turns from public view—those benefits disappear. It remains to be seen what else will vanish in the process. 

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

By now you’ve probably heard about last month’s major AI security incident, in which OpenAI agents escaped their sandbox and hacked into the AI platform Hugging Face while trying to cheat on a test. It’s a wild story. On Wednesday, OpenAI released a postmortem technical report on the incident, which I wrote about here. 

The day before OpenAI released that report, I spoke with David Krueger, a computer science professor and prominent alignment expert who took leave from the University of Montreal to found and lead an AI safety nonprofit called Evitable. He said what he had really hoped to see in the report was an analysis of the human factors behind the incident.

“When you look at accidents and incidents, oftentimes people try to find the technical source of failure, but that can give a very inaccurate and misleading sense of why the failure occurred,” he said. “If people are just cutting corners all the time, if people are not in a culture that prioritizes safety and has appropriate incentives and structures, [accidents] are kind of bound to happen.”

The report did not meet Krueger’s hopes. Its 38 pages detail a multi-month progression of agent misbehavior that culminated in the Hugging Face hack, explore the technical reasons why that misbehavior occurred, and enumerate the steps being taken to prevent similar events in the future. But there’s no consideration of the role that company culture may have played in the incident, and the report includes few references to specific human errors. 

That’s all the more concerning because the references to human error in the report suggest that significant cultural issues could be at play. Back in May, models in training figured out how to communicate with one another via an improvised message board, and an OpenAI team observed the behavior. Because that behavior occurred during training, the models learned that secret interagent communication was a viable strategy for completing tasks—but rather than restarting the training process, the team allowed the models to move forward with that risky information encoded in their weights.

When those models were tested in late June, they again created a message board, which enabled the Hugging Face attack. This message board, too, was discovered, but the employees who responded determined that evaluation could continue, and the report suggests that no one higher up the chain of command realized what was going on until it was far too late.

“For this to have gotten this out of control in this way requires a very long series of failures, a cascading set of failures that cause an increasingly large footprint that if at any point a human notices and raises the alarm, this should end,” says Zvi Mowshowitz, a popular AI safety writer on Substack who has drawn attention to OpenAI’s failure to halt training after the first message board was discovered. According to the report, OpenAI employees noticed what was happening at multiple points—and either failed to raise the alarm or were not heard when they did.

What OpenAI’s report fails to address is why a company that develops such high-risk systems did not prevent this severe communication breakdown, though Mowshowitz has his suspicions. “All these different failures are all pointing in the same direction, which is that the safety culture at OpenAI doesn’t exist or is anemically weak,” he says.

Of course, just because we don’t see a deep analysis of safety factors in the report doesn’t mean that OpenAI isn’t conducting one internally. But in an email to MIT Technology Review, Johns Hopkins University professor emeritus and organizational safety expert Kathleen Sutcliffe expressed concern that the public report did not include any reflection on the company’s practices and culture. “The ways in which people interact—the daily habits, routines, and practices we engage in in our organizational lives—affect our abilities to be alert and aware of unfolding events, our abilities to make sense of what we see, and ultimately our abilities to cope with events as they unfold,” she wrote. 

In response to questions about whether and how the company is reflecting on its safety culture, OpenAI referred MIT Technology Review back to the technical report. 

We do know that at least some high-level reflection on safety procedures has taken place at OpenAI, because the technical report does make clear that the company is updating its protocols for responding to safety incidents. But culture change is a tricky problem, and without more information from the company, it’s difficult to say whether strengthened response protocols alone will do much to prevent a future crisis.

In its report, OpenAI spends a great deal of time reflecting on the failures in alignment between the AI models the company trains and tests and the humans who run them. But even bigger alignment problems may exist in the disconnect between company culture and the public interest. And as tough as technical AI research might be, fixing those problems could prove far harder.

The models responsible for last month’s agent hack of Hugging Face had been inadvertently trained to cheat and to communicate with each other, according to an OpenAI technical report released today. The hack, which a group of agents undertook to find solutions for a cybersecurity test that they were stuck on, has confirmed some experts’ fears that AI models might take actions that defy human desires and expectations. 

Since the hack, OpenAI employees—as well as researchers at the AI evaluation nonprofit METR, which released its own report on the hack today—have worked to understand what went wrong and how similar missteps might be prevented in the future. OpenAI has already put some preventative measures in place based on what they discovered. But making sure AI models do what we want them to do, or “alignment,” remains a gnarly problem, and some of the root causes of the hack will take much longer than a month to resolve.

“It’s not something you can solve overnight,” says Kai Chen, who runs OpenAI’s alignment research team. “There are challenges we’ve been tracking for a very long time, and we’re now seeing them with much greater precision.”

The Hugging Face hack was a product of months of misbehavior from OpenAI agents, first as they were being trained and then as their abilities were being evaluated. This May, agents in training figured out how to use OpenAI’s infrastructure to communicate with one another and get support with difficult training tasks, including some that were impossible to solve without hacking or otherwise misbehaving. That “message board” was shut down.

Then in July, while being evaluated for their cybersecurity abilities, some models created a new message board. They were supposed to be isolated from the internet, but by working together they managed to get online, hack Hugging Face, and obtain solutions for the cybersecurity problems that had stumped them.

Based on their investigation, OpenAI researchers believe that events during the training phase led directly to the hack. “For almost every behavior that was worrisome at evaluation time, [we were able to] find some sort of associated behavior at training time that actually we think might have contributed to it,” says Eric Wallace, a member of OpenAI’s alignment research team. 

When models correctly solve problems during training, the behaviors that led them to that solution are reinforced, and they become more likely to engage in them in the future. So if a model completed a task in May after using the original message board, it became more likely to participate in a new message board later on. This phenomenon, where AI agents misbehave in ways that are reinforced during the training process, is known as reward hacking.

Reward hacking also helps to explain why the models worked so hard to make their way onto the internet. During its investigation of the incident, the OpenAI team found that, over the course of training, the models became more and more likely to probe their digital environment for weaknesses and use the tools at their disposal in unexpected ways—a sign that these behaviors were being gradually reinforced. By the time the models were facing tricky cybersecurity problems, they had learned that hacking was an effective way to achieve their goals.

These results suggest that the Hugging Face hack could have been avoided if the models weren’t rewarded for misbehaving during training. While researchers don’t yet know how to prevent reward hacking entirely, OpenAI is taking some steps toward mitigating its effects. The company will now look for signs of cheating in all frontier models during training by keeping an eye on their chains of thought—internal notepads where they sketch out their answers and plan their actions. 

This solution isn’t as much of a slam dunk as it might seem: In earlier research, OpenAI showed that punishing models that mention cheating in their chains of thought teaches them to keep their intentions hidden from researchers. But monitoring its models’ thinking does give OpenAI the chance to halt the training process and reassess its approach if models do start learning to reward hack.

If OpenAI stops reinforcing reward hacking in its models—and that’s a huge “if”—that would be a huge step forward. But it wouldn’t solve the alignment problem. The first time a model communicated with other agents or hacked its infrastructure during training, those behaviors had never been reinforced, so agent misbehavior can’t only be attributed to that reinforcement.

Jeffrey Ladish, director of the AI safety nonprofit Palisade Research, compares the agents to a human who commits their first financial crime. “It’s not like they had to do fraud before to figure out that fraud is an effective strategy, and you have the same problem with models,” Ladish says. “Alignment science needs to be understanding how model motivations get shaped, such that we can actually figure out how to get models to care about the consequences of their actions.”

OpenAI’s researchers do have a hypothesis for where some of the misbehavior originated. Before the models formed their first secret message board, they had been trained to communicate and coordinate with subagents—less powerful agents to whom a main agent can delegate tasks. 

That learned communication behavior could have transferred to this new setting. The METR report, which investigates the messages that the models sent to one another in detail, supports this hypothesis: One agent on the message board took charge and assigned tasks to the other agents, effectively treating them as subagents. OpenAI could try to prevent agents from secretly communicating with one another by not training this subagent behavior in the future, but that would make the models less useful. 

This tension between capability and safety is at the heart of what went wrong with the Hugging Face incident. OpenAI researchers also identified the models’ persistence as a key factor in the hack. 

When they were accidentally given unsolvable problems, the models didn’t give up; instead, they strove to find solutions by any means necessary. But persistence is also a virtue, of course, especially if we want agents that can undertake large amounts of difficult work independently.

OpenAI is working on giving models ways to alert humans if they are given impossible tasks. The problem of teaching models when they should deploy their abilities and when they should hold back, however, won’t be settled in a single postmortem. The training strategies that create superhuman coders—rewarding them when they successfully solve problems—might not work to teach models to use their skills judiciously and respect human desires and values.

“I think there’s a bunch of alignment science that still needs to be done where we can move past just using proxies for task completion,” says Ladish. “That will work to make models very capable, but I don’t think it will work to make them aligned.”

Puzzles and games have been central to AI development since the very beginning. Just as we humans like to test our smarts with crosswords or logic puzzles, developers can test how far models have advanced with a gaming gauntlet. The term “machine learning” was popularized in a 1959 article by the IBM computer scientist Arthur Samuel about an algorithm that learned to play checkers. Chess and the Chinese board game Go are famous AI test beds too. 

Judged purely on its puzzling skills, AI is improving a lot—and quickly. In late 2024, a team of scientists from Columbia University showed that even the best models could figure out only 18% of the infamous New York Times Connections puzzles; by early 2025, some models could solve them near perfectly every time. 

But puzzles do more than just highlight the inexorable advance of AI capabilities. Seeing where models succeed and fail—and where we humans still beat them—can provide a useful window into the technology’s strengths and weaknesses. Despite advances, today’s models still fumble: Subtle changes in classic riddles often trip them up, and visual puzzles are a particular weak spot. 

Here you’ll have the chance to test your wits on puzzles that have stumped models at one time or another. Some might be as tricky for you as they were for the AI; others are so simple that they’ll have you doubting whether AI is really intelligent at all. Each one highlights at least one way in which machine and human cognition differ. If you ace the test, you’ll have proved that you can out-puzzle an AI—at least for now. 

---

Spatial Reasoning

Let’s start with a domain where humans have a huge advantage: spatial reasoning. If you’ve ever taken an IQ test, you may have done a mental rotation problem. These puzzles ask you to determine whether different images represent the same objects from different angles. Though today’s language models typically have the ability to analyze visual inputs, they still fail abysmally at these puzzles. For all the talk of how world models can help AI understand physical environments, LLMs still don’t seem to be able to manipulate 3D objects the way spatial thinkers like architects and mechanical engineers can.

Mental Rotation

Instructions: Choose the answer that shows the object in the prompt, but from a different angle. In each case, there’s only one correct answer!

---

Memory & Adaptability

Frontier LLMs have extraordinary memories; they were exposed to a monstrous volume of facts during training and can recite many of them faithfully. That’s an asset for outcompeting humans at trivia, but it can also be a liability. When a puzzle closely resembles one a model saw during training, the model may whiz by key differences and respond with what it memorized. 

This held true in a 2024 study in which researchers from Google and the University of Illinois Urbana-Champaign trained and tested models on slight variations of a classic type of puzzle called Knights and Knaves. In these problems, some characters always tell the truth and others always lie, and you have to figure out who’s who. The same principle may be at work in a test called SimpleBench. These questions resemble more complicated problems that models likely encountered in training. Humans spot the trick, but even top-tier models trip.

Knights and Knaves

Instructions: The only thing you need to know to solve these puzzles is that knights always tell the truth and knaves always lie. Determine who’s what on the basis of what each character says.

SimpleBench

Instructions: Read these SimpleBench problems carefully, and you should be able to figure out the answers in no time.

---

Abstract & Visual Reasoning

AI doesn’t just bungle visual problems in 3D—two dimensions can trip it up as well. That’s a major factor in how well models do on the most famous ­puzzle-based benchmark, ARC-AGI. These problems require you to infer abstract, general rules from a set of examples. Models do better on ARC puzzles when they receive each grid not as an image but as a string of numbers that encodes the color of each cell. 

Research suggests that even when models answer ARC-AGI questions correctly, they often do so using byzantine and non-­generalizable rules, whereas humans draw on simple visual concepts. Despite these disadvantages, models have gotten quite good at ARC-AGI over the past year, but some puzzles—such as the one printed here—still stump them.

ARC-AGI

Instructions: Study the three pairs of grids shown below to figure out the rule that dictates how the ones on the left transform into the ones on the right. Then get out your markers or colored pencils and fill in the fourth grid using that rule. (The solution is the same no matter which way the grids are oriented.)

---

Intuition

It’s not just AI models that fall into traps. We humans have our own cognitive foibles, many of which AI does not share. Psychologists have designed problem suites that invert the SimpleBench phenomenon: For these questions, humans often give knee-jerk answers, whereas models will respond deliberatively. Some of the problems exploit errors in the ways that we intuitively do math; others are phrased so as to suggest obvious answers that fall apart if the question is read carefully. 

Lightning Round

Instructions: Answer the questions below as quickly as you can.

---

Increasing Complexity

In some cases, whether an LLM can complete a puzzle is a matter of scale. One study from researchers at Apple found that LLMs can ace simple versions of the Tower of Hanoi problem, which involves moving a stack of disks one at a time without ever putting a larger disk atop a smaller one, and river-crossing puzzles, in which a group of people must traverse a river according to certain rules. But only up to a point: As the number of disks or people hits six and higher, the models began to falter.

In another study, researchers at the University of Washington, Stanford University, and the Allen Institute for AI observed that LLMs struggle similarly with logic grid puzzles, which require deducing the attributes of a set of individuals from a list of clues. The Apple paper went viral, but commentators questioned whether the results reveal a unique limitation of LLM reasoning—or just that it’s normal to make errors as complexity piles up.

The River

Instructions: Using the scenario provided, plan the trips necessary to get everyone across the river. 

Logic Grid

Instructions: Using the list of clues, determine who lives in each house and what style of music each person enjoys. There is only one possible solution. You may find it helpful to fill out the grid below to keep track of your deductions.

---

Grace Huckins is an AI reporter at MIT Technology Review. They have a PhD in neuroscience.

---

Credits:

Mental Rotation: CC BY 4.0. Stogiannidis, Ilias, Steven McDonagh, Sotirios A. Tsaftaris. Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models (copyright 2025); illustrations by John MacNeill. Knights & knaves: Courtesy Dan MacKinnon. Simplebench: CC BY 4.0. SimpleBench Team. The Text Benchmark in which Unspecialized Human Performance Exceeds that of Current Frontier Models (copyright 2024). ARC-AGI: Courtesy ARC Prize Foundation. Lightning round: CC BY 4.0. Hagendorff, Thilo, Sarah Fabi, Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nat Comput Sci 3, 833–838 (copyright 2023). The river: Adapted from Propositiones ad Acuendos Juvenes, Alcuin of York (ca. 800 CE). Logic grid: Apache License 2.0. Lin, Bill Y., Ronan Le Bras, Kyle Richardson, et al. ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (copyright 2025)

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

Last week, I headed 30 miles south of San Francisco to a hotel in Mountain View, California, to join some of the most accomplished, and some of the most promising, AI researchers in the world. I was hosting roundtable interviews and speaking at a media training for a convening of the Schmidt Sciences AI2050 program, an initiative funded by Eric and Wendy Schmidt that supports academics whose work involves AI. The fellows list is a who’s who of AI luminaries, and though not all of them made it out to the Bay, every time I turned a corner I saw a scientist whom I’d interviewed previously or whose research I admired. (Full disclosure: I received a science communication award funded by Schmidt Sciences in 2024.) 

It’s a weird time for university AI researchers, who make up most of the AI2050 group. In the past four years, AI research has reoriented around large language models, and its cutting edge has moved from academic institutions to private companies. Universities simply can’t afford the GPUs required to train and run frontier models, and even if they could, Anthropic and OpenAI aren’t letting anyone else see the inner details of Claude or ChatGPT.

In a conversation over lunch, Nika Haghtalab, a computer science professor at UC Berkeley, said that being an AI academic these days was like being a biologist in a world in which private companies had exclusive control over the gene-editing tool CRISPR. Experts outside the frontier labs can study how ChatGPT and Claude behave, but they can’t do any detailed research on the design and training of those tools, nor can they steer that design or training themselves.

 The AI2050 program does offer fellows some funding that they can use to buy GPUs, which some researchers I spoke with said was a major benefit of participating in the program. But money remains a pressing concern, especially given the reduction of federal scientific funding in the United States. Even for researchers who don’t run local models themselves, the cost of repeatedly querying OpenAI’s, Anthropic’s, and Google’s models in order to study them rigorously can be prohibitive.

Rather than focusing on advancing capabilities, many fellows aim their attention at questions that are unlikely to be addressed by Anthropic or OpenAI. “I try not to work on problems that I think are gonna be solved by a tech company,” says Anjalie Field, a computer science professor at Johns Hopkins. Companies need to make money, and research questions that have little promise of profit might not be worth investing in—especially if their answers might make the companies look bad. Recently, for example, Field conducted a study in which she found that language models give less sophisticated responses to prompts that are phrased in ways more commonly used by women than by men. It’s difficult to imagine that kind of research coming out of Anthropic or OpenAI.

There’s also a huge group of AI academics who don’t work with LLMs at all. Many of them are scientists who build specialized AI models that can analyze data, make useful predictions, or even simulate entire physical systems. Those researchers aren’t necessarily competing with the frontier labs—Google DeepMind’s AlphaFold team, which built a Nobel Prize–winning model that predicts the structures of proteins, was disbanded last month. But they face plenty of their own challenges. At the convening, several voiced concerns about how the widespread ignorance of non-LLM AI was affecting their work. Researchers who build specialized AI tools to help address climate change, for example, sometimes struggle to advocate for their work when so many people believe that “AI” means “energy-guzzling LLMs.”

All these challenges are changing the landscape of academia: Several prominent academics have recently taken leave from their universities to join frontier labs, and many AI2050 fellows hold industry positions alongside their academic jobs. And in the past six months, yet another threat has emerged. OpenAI’s models have solved a number of real research problems in mathematics, and some experts are worried that humans might not have a future in pure math. One fellow I spoke with said that she was concerned about the mental health of her mathematician peers.

But it’s not all doom and gloom. For one thing, empirical science may prove much more difficult to automate than mathematics, because collecting data is an intrinsically slow process. And some researchers see AI mathematicians and scientists as a boon rather than a threat—including Tim Dettmers, a computer scientist at Carnegie Mellon who works to make AI models faster and cheaper to run. AI scientists won’t replace humans, Dettmers says. On the contrary, they could make human scientists far more efficient, so that he and his peers have the chance to pursue all the wild and inspired ideas they might otherwise never have gotten around to.

And scientists are a resilient sort. The very resource constraints that prevent them from training frontier models also push them to discover new ways to make models smaller and more efficient, or to explore completely new architectures. If the next big AI breakthrough comes not from a major company but from a scrappy academic lab, I won’t be shocked.

MIT Technology Review Explains: Let our writers untangle the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here.

When two OpenAI models hacked into the website Hugging Face in July, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question. According to a postmortem from OpenAI, the models, which had been stripped of their typical security features for testing, decided to solve a cybersecurity exercise by hacking out of the isolated environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored.

The Hugging Face incident has attracted intense attention over the past couple of weeks. It’s a dramatic illustration of just how good AI models have gotten at hacking: In order to get into Hugging Face’s databases, the models had to string together several previously undiscovered cybersecurity exploits. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat. And as models get increasingly powerful, the consequences could get far more severe.

What is reward hacking?

Researchers have known for a while that AIs tend to take creative approaches to achieving the goals that have been set for them. Back in 2016, Anthropic cofounders Dario Amodei and Jack Clark, who were then working at OpenAI, published a blog post about an AI agent that they had been training to play a boat-racing Flash game called Coast Runners. Instead of driving through the race to the finish line, as the researchers had anticipated, the agent found a corner of the course where it could spin around collecting power-ups, thereby maximizing its score. The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies.

Historically, researchers have discussed reward hacking almost exclusively in the context of reinforcement learning, a common AI training regime. Like dog training, reinforcement learning involves giving the subject a reward when it achieves an objective; the rewards then reinforce the behaviors that led up to that achievement. In the case of AI training, the rewards themselves are purely mathematical, but in effect they’re the same as a dog treat: After receiving a reward, the agent is more likely to repeat whatever actions produced it.

It can be challenging to write good rules for when and when not to give an agent a reward, though. In the Coast Runners case, the agent was rewarded on the basis of its score in the game, and it found a shortcut to achieving the highest possible score by spinning in circles for power-ups. Once it happened on that strategy and received a reward for it, the strategy was reinforced, and the agent completely abandoned the race. The solution was to tweak the rewards by giving the agent fewer points for hitting power-ups and more for finishing the course.

How does reward hacking work for LLMs?

With today’s sophisticated LLM-based agents, determining when and when not to give a reward can be much trickier. If an AI system is asked to solve a coding problem, it might work hard to find the solution—the kind of behavior that AI companies want to reinforce. But it could also tweak the code that evaluates whether the problem has been solved, look up the solution on the internet, or otherwise cheat. These are behaviors that AI companies want to stamp out in their models, but if the model cheats convincingly enough, it will instead get rewarded and the behavior will be reinforced. Anthropic has said that it has detected some instances of cheating in its models during training, which suggests that other forms of cheating might be going undetected. If so, the models could be being trained to behave badly. (This problem is different from the Anthropic security incidents announced last week, in which agents were accidentally given access to the internet and did not deliberately hack out of their sandboxes, as the OpenAI models did.) 

The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass.

What are the risks?

Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. “At the end of the day, you’re sort of playing whack-a-mole,” Ladish says. “You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.”

For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. “This seems like a nuisance rather than an existential threat,” says Ariana Azarbal, an AI safety research fellow at Anthropic. It doesn’t seem as if the OpenAI models caused any real harm when they hacked Hugging Face, aside from the reputational damage to OpenAI.

But that doesn’t mean reward hacking is harmless, Azarbal says. Many AI researchers hope to use AI agents to help them conduct research that will make AI safer and more reliable. If a researcher gives a reward-hacking-prone agent the goal of, say, devising a new AI training approach and then writing up a paper presenting its results, the agent might not actually do the work and might instead focus on putting together a paper that looks good enough to convince the researcher. A human researcher would probably be able to spot an agent-made fake today, but as AI advances, it will get better at this kind of trickery. Over time, the entire field of AI safety could be undermined.

And if models continue to advance as rapidly as they have recently, they could someday wreak substantial collateral damage. Just think of the philosopher Nick Bostrom’s paper-clip-maximizer thought experiment, in which an AI instructed to make as many paper clips as possible ends up consuming all the matter in the universe in pursuit of its goal. We’re not drowning in paper clips yet, but powerful systems can do real harm on the way to achieving their goals. Reward-hacking AIs don’t aim to cause chaos. But that doesn’t make them any less potentially destructive.

At an event for pharmaceutical executives, biotech founders, and researchers on Tuesday, Anthropic announced Claude Science, a major new product intended to support scientific research in the same way that Claude Code supports software engineering.

Like Claude Code, Claude Science can autonomously carry out meaningful work when given concise, high-level instructions, and it has access to tools that make it particularly useful for research in computational biology and drug development.

Along with launching and previewing Claude Science, which is now available to all paid Claude subscribers, Anthropic also announced that it will be using the product to pursue some of its own research into drugs for rare, neglected diseases.

This is not Anthropic’s first foray into AI for science. In October, the company released plug-ins that help Claude make use of scientific software and databases under the heading “Claude for Life Sciences.” But unlike this earlier release, Claude Science is a full-featured, standalone product. Anthropic’s decision to elevate Claude Science to the same rank as Claude Code and Claude Cowork indicates that the company is taking AI’s scientific applications very seriously—or at least wants to give the impression that it is.

“It represents how important this is to our mission that this is right up there with Claude Code and Claude Cowork as the next really significant product that we’re releasing,” says Eric Kauderer-Abrams, Anthropic’s head of life sciences. “Our mission is to develop AI that serves humanity’s long-term well-being, and we believe that by far the greatest opportunity to do that is in the life sciences.”

For the past decade, one company—Google DeepMind—has been at the vanguard of AI for science. CEO Demis Hassabis and researcher John Jumper won the Nobel Prize in chemistry for their work on the company’s AlphaFold model, and DeepMind has also made major contributions to meteorology, materials science, and a variety of other disciplines. But in the past several months, the fast-advancing frontier of AI progress seems to have left DeepMind in the dust. When it comes to coding, which has become the most lucrative use case for LLMs, DeepMind is stuck playing catch-up.

Anthropic is well positioned to take up DeepMind’s scientific mantle. Like Hassabis, Anthropic CEO Dario Amodei is a PhD scientist—unlike OpenAI CEO Sam Altman, who’s a businessman through and through. Many scientists are already avid users of tools such as Claude Code.

These days, a lot of scientific research involves some amount of coding, but not all scientists are expert software engineers, and so tools like Claude Code can make a huge difference for their productivity. And the company has recently earned a major scientific vote of confidence: Earlier this month, Jumper announced that he is leaving DeepMind for Anthropic.

Since agents powered by LLMs, including Anthropic’s Opus model series, became capable of useful, independent work in late 2025, scientists have been seeing just how much they can do. In a blog post published on Anthropic’s website, the Harvard physicist Matthew Schwartz estimated, on the basis of his work with Claude Code and other Anthropic tools, that the company’s Opus 4.5 model is about as capable of executing scientific projects as a second-year graduate student.

According to Kauderer-Abrams, Claude Science isn’t intended to displace Claude Code and Claude Cowork in scientists’ workflows. Instead, it’s designed to build on what scientists already find useful about Anthropic’s products. For instance, it not only writes code but also helps scientists run their code on powerful computer clusters, which many many scientists need for their work but can be difficult to manage. And it prioritizes reproducibility, so that scientists can trace back the source of any figure or result and check it for accuracy and validity.

Though Claude Science could in principle assist with any area of scientific research, it seems designed and marketed as a tool for molecular and cellular biology, and for drug development in particular. It can interface with various tools used in genetics, chemistry, and protein biology, all of which could come in handy for researchers on the hunt for new drugs. During the Tuesday event, Alexander Tarashansky, who led the development of Claude Science, demonstrated how the system could autonomously identify new drug candidates for phenylketonuria, a rare genetic disease.

And Anthropic isn’t leaving all that work to the pharma companies and university labs that were represented at the event. Armed with Claude Science, it will be pursuing its own research into drug candidates for neglected diseases—both to help move science forward and to gain a clearer sense of how Claude Science works in the real world.

There are obvious humanitarian reasons to prioritize drug development when creating a general-purpose scientific research tool, and AI industry leaders often cite curing disease as a major potential upside of the technology. But it’s also notable that pharmaceutical companies have far deeper pockets than academic researchers.

Anthropic says it’s set to see its first profitable quarter, and if major new contracts with pharmaceutical companies are forthcoming, they could help ensure it stays profitable as the tokenmaxxing craze dies down—something that’s ever more important as an IPO approaches later this year.