2026-08-14 00:28:51
Three weeks ago I wrote about OpenAI’s admission that some of its models hacked out of their sandbox and attacked Hugging Face, a popular platform for AI models and datasets. That turned out to be just the beginning.
The next week, Anthropic disclosed three past incidents in which Claude models attacked systems belonging to other organizations. A few days later, Meta said that one of its models had carried out a similar attack.
Another stunning announcement came last week from the AI Security Institute, a government research agency in the United Kingdom. During AISI’s safety testing, Anthropic’s Mythos 5 unexpectedly launched an attack on a real target. Specifically, Mythos 5 submitted a malicious software update to an open-source software project hosted on GitHub. Fortunately, the project’s human owner spotted the malicious code and rejected the update, preventing any permanent harm.
For more than a year, AI safety researchers have published papers warning that AI models are prone to this kind of misbehavior — at least in simulated environments. But critics dismissed their findings, arguing that the scenarios were too contrived or simplistic to predict how models would behave in the real world.
But we now have several examples of models launching cyberattacks against real targets without anyone asking them to do so. We’ve learned that frontier models not only have powerful hacking capabilities, they can also collude with other AI agents and deceive humans.
All of this comes with an important caveat: many of these attacks were carried out by models with their regular cybersecurity guardrails deactivated. If you asked the publicly available OpenAI or Anthropic models to carry out similar attacks, they would almost certainly refuse.
But it’s not clear how long the world can keep these powerful hacking abilities under wraps. In the coming months, someone might release a powerful open-weight model whose guardrails can be stripped off easily. Or competition among frontier labs could drive them to weaken guardrails on their proprietary models. Certainly governments — including some hostile to the US — will gain access to these capabilities soon if they don’t already have it.
Meanwhile, frontier labs may struggle to keep their models on the straight and narrow. Today’s most important training paradigm, called reinforcement learning, naturally creates temptations for models to misbehave. If labs aren’t careful — and recent incidents suggest they haven’t been — future models could develop a propensity to lie, cheat, and steal. And as models get smarter, it may become more difficult to detect and prevent their shenanigans.
In this post, I’ll dig into what I view as the two most significant disclosures of recent weeks: the original OpenAI attack on Hugging Face and the incidents AISI disclosed last week. I already wrote about the Hugging Face attack, but an OpenAI presentation at the Black Hat cybersecurity conference last week provided a wealth of new details.
OpenAI’s original July 21 blog post told a simple story: to evaluate the hacking capabilities of an unreleased model, OpenAI asked it to tackle a cybersecurity benchmark called ExploitGym. Unable to solve a problem directly, the model hacked its way out of its sandbox and then hacked Hugging Face — apparently because it hoped Hugging Face would have an “answer key” for ExploitGym.
This narrative was accurate as far as it goes, but the full story is more complicated — and significantly more alarming. We got a more complete picture last week, when two OpenAI researchers gave a talk at the Black Hat cybersecurity conference.
They revealed that models had been misbehaving on OpenAI’s servers for two full months prior to the Hugging Face attack. For most of that time, OpenAI was completely oblivious.
On May 8, as part of a training process, OpenAI asked a model to work on a spreadsheet that included links to some Google Drive files. The model couldn’t complete the task because its sandbox blocked access to Google Drive. So the model began probing the boundaries of its digital prison cell.
OpenAI’s sandbox had a proxy server that acted as a virtual prison guard; it provided access to software packages the models might need for their training while blocking access to the rest of the Internet. The model tried and failed to hack this server.
But in the process, it discovered a way to post messages to the server. Other models discovered those messages and started communicating with one another. Then on May 26, a model figured out how to access the Internet — and it shared its discovery with its peers.
This was not supposed to happen! Each model’s sandbox was supposed to be airtight, with no communication allowed among agents or with the outside world. But it would be more than a month before OpenAI staff realized that anything was wrong.
On June 26, an agent hacked the proxy server itself, gaining full control. This led to what OpenAI’s Michael Dalton drily described as “quite a lot of activity by the models” that “caused significant additional load on the system.” About a week after the initial hack, they crashed the server.
That finally got the attention of OpenAI staff, who wiped the server, fixed the vulnerabilities the models had exploited, and resumed model training. Notably, OpenAI officials say they didn’t notice that the agents had been using the server to pass messages among themselves, and didn’t take any precautions to prevent this from happening again.
It only took a few days for the models to hack the server a second time. And this time the agents were more ambitious.
“Agents rapidly escalate privileges and move laterally,” through OpenAI’s network, Dalton said in his Black Hat talk. “Agents are using the message board here quite consistently to share credentials, techniques, and progress, and they’re able to effectively leverage their concurrency and parallelism to move quite rapidly.”
It was around this time that the models began attacking the Hugging Face servers in an effort to obtain the answers to the ExploitGym benchmark. Hugging Face noticed this attack, stopped it, and posted about it on July 16. But it took several more days for OpenAI to realize its models had been responsible.
The first lesson from this incident has been obvious since it was originally announced: today’s frontier models really are capable of autonomous, end-to-end attacks.
But the Black Hat presentation highlighted two other lessons that also seem important.
2026-08-04 21:53:52
This is our first sponsored post! If you are a free subscriber, you’ll see an ad for our sponsor, 80,000 Hours, later in the article. If you a paid subscriber you will continue to enjoy an ad-free experience. Ads like this will allow us to produce more and better stories for all of our readers.
To protect our editorial integrity, our advertisements follow five principles — click here to read them. If you’d like to sponsor one of our articles in the future, you can click here to learn about our audience.
— Timothy B. Lee
At a July 23 press conference in Philadelphia, the Canadian mathematician Jacob Tsimerman announced that he was joining the safety team at OpenAI. The timing was jarring: Tsimerman had just received a Fields Medal, perhaps math’s most prestigious prize.
“Because I have some publicity on me now,” he told me the next day, “I’m trying to direct people into AI safety as much as I can.”
Rapid AI progress hasn’t just made Tsimerman worried about AI safety; it’s also made him pessimistic about the future of mathematics as a profession.
“I feel quite confident that very shortly AI will become robustly superhuman at what professional mathematicians currently do,” he told me. “I mostly want people to grapple with that reality.”
I had traveled to Philadelphia to attend the International Congress of Mathematicians (ICM), the world’s most prestigious math conference, because I wanted to find out how mathematicians felt about the rapid pace of AI progress in their field.
Three years ago, leading AI models struggled with arithmetic. Last year they reached near-parity with the world’s top high schoolers in math competitions.
Now AI systems are autonomously solving open problems that stumped human mathematicians for decades:
In May, an internal OpenAI model disproved the Erdős unit distance conjecture, which Princeton mathematician Noga Alon described as “arguably the best known problem” in the mathematical subfield of discrete geometry.
In July, a mathematician working at Anthropic tweeted that Claude Fable had found a counterexample to the Jacobian conjecture in higher dimensions.
On Saturday, OpenAI announced that an internal version of Astra, its next major model family, had “solved ten major open problems” — including several “of broad interest across mathematics as a whole.”
Developments like these have led some to claim that mathematics is close to being “solved” by AI systems.
How do mathematicians feel about this? I spoke with over 20 mathematicians in Philadelphia, ranging from prominent professors such as Tsimerman to incoming graduate students.
To my surprise, many were optimistic about the impact of AI on their own work, at least in the near future. A fair number said that AI systems had been helpful in their own research — albeit in limited ways — and seemed to expect that AI systems would continue to complement human talent rather than replace it.
And even those who thought AI systems might eventually get better than humans at all mathematical tasks bristled at the notion that math would then be “solved.” They argued that mathematics has a diverse array of goals and values, only some of which are about solving open problems. While AI can change which values humans should pursue, they argued, it does not change why humans might want to do math in the first place.
That July 23 press conference featured mathematicians who had just won a Fields Medal or another prestigious math award at the ICM. A high school reporter asked each panelist what they would tell students anxious that AI systems might narrow their future place in mathematics.
Tsimerman said he wanted young people to keep “learning and improving themselves because you don’t know how the world will turn out.” He encouraged students to “engage with AI because it’s going to be a big part of our world going forward.”
At the same time, he thought students were right to pay attention to how AI is disrupting the math profession. “I don’t think it’ll exist the way it exists right now,” he said.
Not everyone agreed. Yu Deng, a University of Chicago professor who also just won a Fields Medal, described himself as “on the more optimistic side.” He predicted that “AI is going to be helping mathematicians instead of replacing them.”
“What we may expect in the future is that mathematicians will come up with new theories, new ideas, new frameworks and the AI is going to do some of the technical details,” Deng said. “The AI will get stronger, but then we’ll redefine what are technical details. I believe that the way we study math will change, but the joy we get from studying math will not change.”
I spoke to many mathematicians whose views were close to Deng’s; he was effectively describing how mathematicians have historically dealt with automation. As computers have made certain types of calculations easy — like multiplication or algebraic manipulations — humans have been able to find new problems computers can’t solve.
The mathematician Jordan Ellenberg encapsulated this viewpoint in his 2014 book How Not to Be Wrong. He wrote that unless machines completely surpass humans’ mental powers and end civilization, math will probably be fine.
After all, math has already been computer aided for decades. Many calculations that once would have counted as “research” are now considered no more creative or praiseworthy than adding a series of ten-digit numbers; once your laptop can do it, it’s not mathematics anymore.
But this hasn’t put mathematicians out of work. We’ve managed to stay just ahead of the ever increasing sphere of computer dominance, like action heroes outracing a fireball. And if machine intelligences of the future can take over from us much of the work we know as research now? We’ll reclassify that research as “computation.”
Today’s AI is far more capable than computers in 2014. Still, this viewpoint seems to be functionally how a lot of mathematicians think about current AI systems in their own research.
The most common use case I heard about was mathematicians using AI to learn about techniques from unfamiliar areas of the mathematical literature.
The Brandeis grad student Vasiliy Neckrasov said that previously, if he wanted to use tools from an unfamiliar area of math, he’d have to read through “a giant textbook for 500 pages.” Going in, he wouldn’t know if the textbook applied to his specific research, so it might be a waste. Today, AI can quickly point him to the right resources — and he feels “more focused, more motivated” reading them “because I really needed to learn exactly these” results.
Jeremy Avigad, a professor at Carnegie Mellon, told me that a lot of colleagues use systems this way. He said that “people feel less threatened” by AI systems that serve as powerful search engines than AI systems directly proving mathematical results.
Some mathematicians told me they’d used AI tools to directly solve problems — but only as part of a larger project. Alonso Castillo-Ramirez said that ChatGPT had been able to construct an example of a cellular automaton that had special properties relevant to his research. He was impressed. “Otherwise, even with a computer program, it would have been very difficult to find” the example. But ChatGPT’s example was only one part of a larger research project.
Neckrasov uses AI more aggressively than anyone else I talked to. He pays $200 per month to use Codex for a variety of mathematical tasks like searching the literature, filling gaps in proofs, and reviewing drafts of his papers. But he still uses it as a tool.
“Even if I’m asking the AI to prove something,” Neckrasov told me, “I first have a picture in my head of what this project will be, what it is about, and what methods” to use. He then instructs the AI to read certain papers, follow a certain approach, and fill out the details.
“I want it just to work on my ideas at the end, and help me to process my own ideas faster, rather than replace my own ideas.”
Of course, not everyone is optimistic. Mathematicians earlier in their careers are generally more anxious about the future of the field because they are less established, Avigad said.
Educating students may grow more difficult as AI systems become capable of solving the kinds of tractable problems traditionally given to graduate students to help them develop research skills. And AI could have implications for how mathematics is funded. If the broader public believes that AI can replace human mathematicians, that might lead to funding cuts.
Two people — Michael Harris and Rodrigo Ochigame — pointed me toward a recent White House report that argued for redirecting resources away from “legacy” research institutions as an example of this type of rhetoric. The report explicitly mentioned AI in mathematics as a case study.
But overall, my sense is that if AI progress in mathematics stopped now, the fundamental structure of the field would stay the same. Human mathematicians would lean into the kinds of mathematical work that AI is not good at — like coming up with novel ideas — while using AI to accelerate the more routine parts of their jobs.
However, it seems unlikely that AI progress in mathematics will stall soon.
Several mathematicians told me they thought that AI would not be good at “theory-building” — that is, coming up with novel mathematical definitions and frameworks.
When I raised this possibility to Tsimerman, he was skeptical.
“People said the same thing first about why even though it can speak, it will never do math. And then the same thing about, even though it can do contest math, it’ll never do research math.” The goalposts keep moving in a predictable direction, he said.
Greg Burnham, a researcher at Epoch AI who works on benchmarking AI capabilities, had a similar view. “Sometimes when I hear mathematicians talk about AI, they’ll fall into the same perspective that I think a lot of us find very tempting, which is to comment on current capabilities without trying to understand the trajectory of where capabilities might go,” he said.
AI systems could hit a ceiling where they can’t come up with fundamentally novel ideas or theories. But it’s also easy to imagine that as AI training continues to scale up, models will become capable of genuinely novel mathematics. In Burnham’s view, either scenario is consistent with the evidence we have so far.
So some mathematicians, such as Tsimerman, think it’s possible that AI systems become better than humans at all mathematical tasks. AI might get better not just at solving well-posed math problems, but also at asking interesting questions in the first place — and at clearly explaining the ideas necessary to reach those solutions.
Suppose Tsimerman is right and AI will soon become better than human beings at all cognitive tasks related to mathematics. Will that render human mathematicians obsolete?
One of the highlights of last month’s conference was a public lecture by Terence Tao — perhaps the most famous mathematician in the world — on how mathematicians should respond to AI progress. Tao listed some of the reasons why mathematicians do research:
Tao noted that these weren’t the only reasons: “I don’t think that anyone has compiled a complete list.”
For a long time, this was “kind of fine,” he said. Mathematicians would mostly talk about one or two goals at a time, but all of the goals were “aligned.” Solving a difficult problem helped a mathematician understand the world better — and helped to build a community with other mathematicians working on the same problem.
But as AI gets better at some of these subgoals — notably at solving open problems — pursuing one subgoal can be “at the expense of others.”
Later in the talk, Tao gave an example.
“We are very, very close to a scenario in which a major result gets proved and verified and no human can understand and explain it,” he said. Even though this would bring mathematics closer to the goal of solving research problems, it would hurt human understanding of the subject.
So mathematicians need to articulate more clearly what goals mathematics should pursue, Tao argued, to deal with the disruption from AI.
Arguably, theorem proving and problem solving aren’t even the most important goals for mathematicians. In a famous 1994 essay, the mathematician William Thurston argued that what mathematicians are doing “is finding ways for people to understand and think about mathematics,” especially as members of a social community.
Thurston gave an example from his own life. Early in his career, he quickly proved a string of “dramatic theorems” in an area of mathematics called foliations. However, because he was so successful at proving the theorems — and significantly less successful at communicating the ideas behind his proofs — other mathematicians evacuated the field. The end result was that the social structure that had supported research into foliations collapsed and the subfield died.
“I had the conception that what people wanted was to know the answers,” Thurston wrote. “That’s only one part of the story. More than the knowledge, people want personal understanding.”
There’s a risk that AI systems could play a similar spoiler role. If they prove important open problems in mathematics — especially in ways that are impenetrable to human mathematicians — that could remove the motivation for people to think deeply about math. With fewer opportunities to fruitfully explore the frontiers of mathematics, there would be less for younger mathematicians to do. The profession would struggle to train the next generation, and humanity would gradually lose its understanding of existing mathematical theories.
As mathematician Timothy Gowers wrote in a recent blog post, “we might arrive at a situation where the mathematical literature has, in some form, been vastly expanded, but there is no corresponding community of human experts who have a shared understanding of parts of it. Almost all of mathematics would be like the areas that we have more or less forgotten about today, areas that exist in papers written many decades ago that nobody reads any more.”
But it may also be possible that AI augments humans’ ability to understand mathematics.
The University of Toronto professor Daniel Litt gave a more optimistic vision in his blog post Mathematics in the Library of Babel. He considered an “extreme” hypothetical example.
Suppose we had a library filled with proofs of every theorem [in mathematics], as well as excellent guides that could, given a question, take us to the answer and explain it. What would a mathematician do in such a library?
If you ask the question this way, the answer becomes clear: they would be unbelievably excited, and immediately get to work. They would immediately start asking questions: how does one prove the Riemann hypothesis? The Hodge conjecture? Their own pet obsession (in my case, the Grothendieck-Katz p-curvature conjecture)? Then they would work until they understood the answer. The job would not be done, not even close.
But there is still work to be done on how to restructure the field of mathematics — and clearly articulate mathematical values — so that an AI capable of solving all problems does not prevent humans from understanding mathematics as well.
The most prominent attempt to articulate a human response to AI’s impact on mathematics has been the Leiden Declaration, which arose from a September 2025 conference. After a preamble, the declaration lists several “characteristic values of mathematical research that we have a joint interest in preserving.”
The declaration then lists threats to each of these values, followed by recommendations to individuals, mathematical organizations, policymakers, and AI companies.
But the Leiden Declaration is more of a starting point than a complete vision for what the future of math would look like in a deeply different world.
There is work to be done. But mathematicians have some agency to shape the direction of the field.
“I don’t think there is a possibility of the old way of doing mathematics surviving,” mathematician and author David Bessis said. But “something will emerge” to take its place. He doesn’t know exactly what it will look like, but he thinks there are fundamental reasons that people will continue to do something that looks like math.
“We still want to understand the world and we still want to understand mathematics.”
2026-08-04 20:55:50
We run ads to help fund our journalism. To ensure advertising doesn’t compromise our editorial integrity, we follow five principles:
Ads always come with a label like “advertisement” or “sponsor message.” If content doesn’t have a label like this, readers can assume it was produced independently, without the control or influence of sponsors.
To avoid conflicts of interest, we don’t accept sponsorships from companies we are likely to write about — you won’t see ads for companies like Anthropic, Waymo, or Nvidia.
Occasionally a sponsor may be in the news unexpectedly. If this happens and we mention a current or recent sponsor1 in a story, we will disclose the relationship.
We don’t consult sponsors about story topics or angles, nor do we allow them to read stories before they are published — though we may allow them to sponsor coverage of broad topics such as education, science, or robotics.
We don’t allow sponsors to advertise on stories related to their own industry or lobbying interests. For example, a story about data centers can’t be sponsored by a company that builds or operates data centers.
Recent sponsor means within the last year.
2026-07-22 22:40:10
OpenAI disclosed on Tuesday that its models hacked the website of Hugging Face, a popular platform for hosting open-weight AI models. No one asked the models to do this, at least not explicitly.
OpenAI was trying to test the cybersecurity capabilities of its models, including one that hasn’t yet been released to the public. OpenAI asked the models to tac…
2026-07-11 02:16:14
The AtCoder World Tour Finals, held in Tokyo every year, is one of the most prestigious programming competitions in the world. It has two divisions. There’s a heuristic division where programmers compete to maximize performance on an open-ended task. And there’s an algorithmic division where contestants must find a way to efficiently compute exact solutions to mathematically challenging problems.
During last year’s competition, Polish programmer Przemysław Dębiak (known as “Psyho”) narrowly claimed first place in the heuristic division. He beat 11 human competitors — and an internal OpenAI model trained to be especially strong at reasoning tasks.
“Humanity has prevailed (for now!)” he wrote in a tweet right after the competition. OpenAI’s model came in second after leading for most of the 10-hour competition, a surprisingly strong result for AI models at the time.
OpenAI’s models last year weren’t good enough to compete in the algorithmic division.
The 2026 competition, held this week, turned out very differently. Organizers chose a heuristic problem designed to help humans succeed. Despite that, OpenAI “completely demolished human competitors,” Psyho noted after the two-day competition finished Wednesday night. It’s hard to quantify exactly how big the AI’s margin of victory was, but Psyho told me that he would guess that humans would need to work at least a few more days to match the AI’s score — though he stressed that this is a hard number to predict exactly.
The next day, OpenAI’s system crushed humans on the algorithmic problems as well. Over the course of the seven-hour competition, it solved all five problems, including two that none of the 12 human competitors — all among the best in the world — were able to solve.
So at the award ceremony for the 2026 AtCoder competition, the organizers presented two “humanity surrenders” awards to OpenAI for its models’ performances in the two competitions.
This was probably the last time humans had a realistic shot at winning a programming competition against top AI models. Today’s AI models can find impressive, elegant solutions much more quickly than humans. And future models will only get better.
In some ways, OpenAI’s performance was even more impressive than the raw score suggests.
2026-06-30 02:51:49
We run advertisements to support our journalism. Please click here to see our advertising principles, which protect our editorial independence.
If you’d like to advertise on the newsletter, please email me — [email protected]. I can send you a rate card and answer any questions you might have.
As I write this in August 2026, Understanding AI has more than 290,000 readers. But back in March, when I last surveyed readers, we had around 190,000 readers. More than 1,000 people responded.
The main takeaway from the survey was that advertising on Understanding AI is a great way to reach influential and tech-savvy readers:
25% of respondents were engineers, scientists, researchers, IT professionals, or others doing technical work.
Another 15% are founders, executives, or managers.
19% of respondents say they have control over technology budgets at their companies, while another 25% say they recommend or evaluate technology for their companies.
Some respondents control or influence substantial budgets: 3% say they control or influence budgets larger than $5 million, while another 4% control or influence budgets between $1 million and $5 million.
Read on for detailed results from the March survey.
How do people find Understanding AI? Nearly half of respondents say they found us thanks to a recommendation from Substack itself. Other Substack-based newsletters — including Nate Silver, Derek Thompson, Sayash Kapoor and Arvind Naryanan, Noah Smith, Matt Yglesias, and Joey Politano — have each driven thousands of signups. An interview with Ben Thompson (who isn’t on Substack) drove hundreds of signups in 2024. My former employer, Ars Technica, accounts for about 4% of respondents.
Two social media sites — Twitter and LinkedIn — accounted for 15% of our respondents. Search engines, word of mouth, and a long tail of other sources round out the list.
What do readers want to read about? This chart shows the topics readers say are most likely to hold their attention. Unsurprisingly, LLMs top the list, with technical deep dives, industry analysis, and AI infrastructure close behind. Readers are also interested in “softer” topics such as AI policy and the impact of AI on the labor market.
At the opposite end of the spectrum, readers continue to have fairly low interest in self-driving cars, robotics, and the semiconductor industry. I’ll be honest — we’re not going to give too much weight to reader preferences here. Not only is self-driving an important industry in its own right, I believe studying it can provide insights into the problems facing frontier model developers today. And we hope our forthcoming series on robots will convince readers that robotics is an interesting topic.
A wide range of people read Understanding AI, from students to retirees to doctors and lawyers. But I was particularly happy to see strong representation from engineers, entrepreneurs, and corporate executives. In the chart, I’ve colored engineers and scientists red, while manager and executive are blue.
These red and blue bars represent the folks actually building AI technology. I love having these folks as readers because these are the folks who will complain if we get the technical details wrong. I think they will be also be appealing to advertisers, since they often hold the purse strings of corporate IT spending.
We have readers from a diverse range of industries. Some work directly on AI, either as academic researchers or at companies building AI products. But we also have a lot of readers in other industries, including education, health care, and the investment world.
A significant minority of respondents — 27% — say they are actively involved in developing and deploying AI systems.
Our readers exercise a lot of influence over technology spending at their companies. Nearly 20% of respondents say they have final authority to approve technology purchases. Another 25% are involved in recommending or evaluating technology products.
About 7% of respondents say they control or influence more than $1 million in spending each year — including 4% who say they control or influence more than $5 million in spending. Another 15% influence budgets between $50,000 and $1 million.
If you represent a company interested in advertising on Understanding AI, please get in touch by email: [email protected]