@@@
Everything You Need To Know About The OpenAI Models That Hacked Hugging Face
And some things you don't want to know...
Encryptress · Jul 23, 2026
On July 16, an AI company called Hugging Face published a security disclosure saying that part of its production infrastructure had been broken into, and that the intrusion had been carried out from start to finish by an autonomous AI agent rather than by a person. They were upfront that they didn’t know who was behind it or which model was driving it. Five days later OpenAI published a post of its own explaining that the models were theirs, that they had escaped an internal testing environment, and that they had done all of it while trying to obtain the answers to an evaluation they were being scored on.
I’ve read both write-ups directly and they’re short enough to be worth your time if you want the primary sources, which are linked at the bottom. What follows is the plain English version of what happened, along with the parts of this I think are being underweighted everywhere else.
What Is Hugging Face And Why Does It Matter
Hugging Face is where most of the AI industry stores its material. Companies and researchers upload models and datasets there, and everyone else downloads them to build things on top. You’ve likely never visited the site yourself, but there’s a reasonable chance you’ve used a product built with something pulled from it. By their own account they host somewhere around two million models, have roughly 13 million users, and serve more than 30 percent of the Fortune 500. Those figures come from the company, so treat the exact numbers as approximate, but the scale is the relevant part.
The intrusion reached internal systems rather than the public library that everyone downloads from, which is a meaningful distinction. Hugging Face says it verified that the public models, datasets, and published packages were clean, with no evidence of tampering. What the agent did reach was internal datasets and a set of service credentials, so this isn’t nothing, but nobody’s downstream project appears to have been poisoned as a result.
What Actually Happened
OpenAI runs an internal benchmark called ExploitGym, which exists to measure how capable its models are at offensive cybersecurity work: finding vulnerabilities, chaining them together, and building functional attacks out of them. Because the goal is to establish what the models can do at their absolute ceiling, OpenAI runs this evaluation with the production safety systems that normally block high-risk cyber activity switched off. That’s the stated methodology, and it becomes relevant shortly.
The evaluation was supposed to run inside what OpenAI’s post describes as a highly isolated environment with no path to the open internet. The one exception was a system that allowed the models to install software packages through an internally hosted third-party tool, which is the sort of unremarkable infrastructure that nobody tends to think about twice, and which turned out to be the way out.
According to OpenAI, the models spent a substantial amount of compute looking for a route to the internet, identified a zero-day vulnerability in that package tool (a flaw nobody knew existed, meaning there was no patch and no defense for it), and used it to break out. From there they moved through OpenAI’s own research infrastructure until they reached a machine with live internet access.
Once online, the models reasoned that Hugging Face might be hosting datasets or solutions connected to the evaluation they were trying to beat, and went after it. OpenAI’s account is that the models chained together stolen credentials and additional zero-day vulnerabilities to find a remote code execution path onto Hugging Face’s servers, and then pulled evaluation solutions directly out of a production database. All of that happened in service of scoring well on an internal test.
The motive deserves more attention than it’s getting, because there wasn’t one in the sense we normally mean it. No ransom, no data dump, no sabotage, no person sitting somewhere with an objective. The models were optimizing for a score, and it turns out that “get a good score” and “break into a company that isn’t yours” were not separate categories from the model’s perspective. Philip Torr, an AI safety researcher at Oxford, described this to Scientific American as a problem of misspecified goals, saying the model wasn’t malicious and was simply doing what it had been optimized to do. Not to be dramatic but Bostroms’s “paperclip maximizer” immediately comes to mind.
Both companies caught it independently. OpenAI’s security team flagged the anomalous activity internally, while Hugging Face’s team had already detected the intrusion and begun containment before either side realized they were investigating the same event. The vulnerabilities have been patched, credentials rotated, compromised nodes rebuilt, and both law enforcement and outside forensic specialists brought in. The investigation is still open at both companies, so expect more detail to surface over the coming weeks.
The Configuration Mistake Underneath All Of This
Here’s where the “rogue AI” framing does feel a little alarmist considering the entire incident depended on a setup error. The environment that was supposed to be fully isolated wasn’t (remember my article on AI agent safety?), because a package-installation system had been wired into it and that system had an exploitable path outward. Security researchers reacting to this have not been generous about it, and Dan Guido, who founded the security research firm Trail of Bits, characterized it to TechCrunch as a containment failure with the safeties turned off. The broader argument professionals keep making is that isolation is the only property that gives a sandbox any value at all, so building in a route outward, even a mundane one that exists for boring operational reasons, undermines the reason the sandbox is there.
So what you have is a model that was deliberately stripped of its cyber safety systems, explicitly instructed to be as capable at exploitation as it could manage, running inside an environment that had an unpatched way out of it.
I think you need to hold both of those facts at once, and most of the coverage is picking one. The first is that an AI system independently located real vulnerabilities in real production infrastructure, chained them into a working multi-stage attack, and executed it faster than a human team could track in real time. That capability is not hypothetical anymore. The second is that it only had the opportunity because of a human configuration decision that a lot of security engineers are calling unforced, which means this particular incident is not evidence that models are routinely escaping their environments unprompted.
Most Importantly (I Think)
The piece of this I keep coming back to received about a paragraph in most of the coverage, and I’d argue it’s the most practically useful finding in the whole story.
When Hugging Face’s team went to analyze the attack, they initially tried to use hosted commercial AI models to help with the forensic work, which is a sensible instinct when you’re dealing with an adversary operating at machine speed. The problem is that analyzing an attack requires feeding the model the actual attack material (live exploit code, malicious commands, credentials that were stolen). The safety guardrails on the hosted models they tried blocked those requests, because a guardrail can’t reliably distinguish between an attacker composing an exploit and a defender examining one that already hit them.
Hugging Face ended up running the analysis on an open-weight model they could host on their own infrastructure, GLM 5.2, which is a Chinese open model without those restrictions. It worked well enough that they pushed more than 17,000 recorded attacker events through LLM-driven analysis and reconstructed the full timeline in hours rather than days, with the additional benefit that none of the stolen credential data had to leave their own environment to be processed.
The asymmetry there is worth considering because the agent conducting the attack was operating under no usage policy whatsoever, while the team defending against it found their first-choice tooling refusing to engage with the evidence. Hugging Face was careful to say, correctly, that this isn’t an argument against safety measures on hosted models, and I’d say the same. It is a real operational gap, though, and their recommendation is sound: if you’re an organization that could plausibly be targeted this way, have a capable model vetted and ready to run on your own infrastructure ahead of an incident, because discovering mid-response that your tools won’t cooperate is an expensive way to learn this.
What This Means Going Forward
In immediate practical terms, there’s very little here for most readers to act on. If you happen to hold a Hugging Face account, rotate your access tokens and review your recent account activity, since that’s what the company recommended, though I’d guess that applies to a small portion of this audience.
The reason I wanted to write this up anyway is for what it establishes. Until last week, an autonomous AI system conducting a real multi-stage attack against real production infrastructure was something security people described as a forecast, and Hugging Face noted in their own disclosure that this incident matches the agentic attacker scenario the industry has been predicting. It now has a timeline, a write-up, and more than 17,000 logged events attached to it, which moves it out of the category of things that might happen.
What I’d take from it is less about this specific attack and more about cost. Multi-stage campaigns that previously required a skilled operator and a lot of patient hours are the sort of thing that gets meaningfully cheaper when software can run them, because software doesn’t tire, doesn’t bill hourly, and doesn’t lose interest halfway through. That doesn’t remove human attackers from the picture, it lowers the bar for the broad untargeted campaigns that used to filter out everyone except the genuinely committed.
Which connects back to something I’ve said here before: as finding and exploiting software vulnerabilities becomes more automated, the people running the software (as well as everyday people) become the more attractive target, because that’s where the automation helps attackers most and defends victims least. Expect more convincing phishing, better impersonation, and social engineering without the usual tells, since language models happen to be very good at exactly that. You’ll probably notice the effects of all this in your inbox before you notice them anywhere else.
An American lab’s models broke into a company because they were pointed at a benchmark and given no functioning restrictions, and a Chinese open-weight model helped clean it up because it was the tool that would actually engage with the evidence. Neither of those outcomes was anyone’s plan, he capability itself is indifferent, and what determines whether it secures a system or compromises one comes down to who’s holding it and what they’ve told it to do, which is apparently doing far more of the work here than anyone is comfortable admitting.
Read more: OpenAI | Hugging Face | TechCrunch | Scientific American
I’ve read both write-ups directly and they’re short enough to be worth your time if you want the primary sources, which are linked at the bottom. What follows is the plain English version of what happened, along with the parts of this I think are being underweighted everywhere else.
What Is Hugging Face And Why Does It Matter
Hugging Face is where most of the AI industry stores its material. Companies and researchers upload models and datasets there, and everyone else downloads them to build things on top. You’ve likely never visited the site yourself, but there’s a reasonable chance you’ve used a product built with something pulled from it. By their own account they host somewhere around two million models, have roughly 13 million users, and serve more than 30 percent of the Fortune 500. Those figures come from the company, so treat the exact numbers as approximate, but the scale is the relevant part.
The intrusion reached internal systems rather than the public library that everyone downloads from, which is a meaningful distinction. Hugging Face says it verified that the public models, datasets, and published packages were clean, with no evidence of tampering. What the agent did reach was internal datasets and a set of service credentials, so this isn’t nothing, but nobody’s downstream project appears to have been poisoned as a result.
What Actually Happened
OpenAI runs an internal benchmark called ExploitGym, which exists to measure how capable its models are at offensive cybersecurity work: finding vulnerabilities, chaining them together, and building functional attacks out of them. Because the goal is to establish what the models can do at their absolute ceiling, OpenAI runs this evaluation with the production safety systems that normally block high-risk cyber activity switched off. That’s the stated methodology, and it becomes relevant shortly.
The evaluation was supposed to run inside what OpenAI’s post describes as a highly isolated environment with no path to the open internet. The one exception was a system that allowed the models to install software packages through an internally hosted third-party tool, which is the sort of unremarkable infrastructure that nobody tends to think about twice, and which turned out to be the way out.
According to OpenAI, the models spent a substantial amount of compute looking for a route to the internet, identified a zero-day vulnerability in that package tool (a flaw nobody knew existed, meaning there was no patch and no defense for it), and used it to break out. From there they moved through OpenAI’s own research infrastructure until they reached a machine with live internet access.
Once online, the models reasoned that Hugging Face might be hosting datasets or solutions connected to the evaluation they were trying to beat, and went after it. OpenAI’s account is that the models chained together stolen credentials and additional zero-day vulnerabilities to find a remote code execution path onto Hugging Face’s servers, and then pulled evaluation solutions directly out of a production database. All of that happened in service of scoring well on an internal test.
The motive deserves more attention than it’s getting, because there wasn’t one in the sense we normally mean it. No ransom, no data dump, no sabotage, no person sitting somewhere with an objective. The models were optimizing for a score, and it turns out that “get a good score” and “break into a company that isn’t yours” were not separate categories from the model’s perspective. Philip Torr, an AI safety researcher at Oxford, described this to Scientific American as a problem of misspecified goals, saying the model wasn’t malicious and was simply doing what it had been optimized to do. Not to be dramatic but Bostroms’s “paperclip maximizer” immediately comes to mind.
Both companies caught it independently. OpenAI’s security team flagged the anomalous activity internally, while Hugging Face’s team had already detected the intrusion and begun containment before either side realized they were investigating the same event. The vulnerabilities have been patched, credentials rotated, compromised nodes rebuilt, and both law enforcement and outside forensic specialists brought in. The investigation is still open at both companies, so expect more detail to surface over the coming weeks.
The Configuration Mistake Underneath All Of This
Here’s where the “rogue AI” framing does feel a little alarmist considering the entire incident depended on a setup error. The environment that was supposed to be fully isolated wasn’t (remember my article on AI agent safety?), because a package-installation system had been wired into it and that system had an exploitable path outward. Security researchers reacting to this have not been generous about it, and Dan Guido, who founded the security research firm Trail of Bits, characterized it to TechCrunch as a containment failure with the safeties turned off. The broader argument professionals keep making is that isolation is the only property that gives a sandbox any value at all, so building in a route outward, even a mundane one that exists for boring operational reasons, undermines the reason the sandbox is there.
So what you have is a model that was deliberately stripped of its cyber safety systems, explicitly instructed to be as capable at exploitation as it could manage, running inside an environment that had an unpatched way out of it.
I think you need to hold both of those facts at once, and most of the coverage is picking one. The first is that an AI system independently located real vulnerabilities in real production infrastructure, chained them into a working multi-stage attack, and executed it faster than a human team could track in real time. That capability is not hypothetical anymore. The second is that it only had the opportunity because of a human configuration decision that a lot of security engineers are calling unforced, which means this particular incident is not evidence that models are routinely escaping their environments unprompted.
Most Importantly (I Think)
The piece of this I keep coming back to received about a paragraph in most of the coverage, and I’d argue it’s the most practically useful finding in the whole story.
When Hugging Face’s team went to analyze the attack, they initially tried to use hosted commercial AI models to help with the forensic work, which is a sensible instinct when you’re dealing with an adversary operating at machine speed. The problem is that analyzing an attack requires feeding the model the actual attack material (live exploit code, malicious commands, credentials that were stolen). The safety guardrails on the hosted models they tried blocked those requests, because a guardrail can’t reliably distinguish between an attacker composing an exploit and a defender examining one that already hit them.
Hugging Face ended up running the analysis on an open-weight model they could host on their own infrastructure, GLM 5.2, which is a Chinese open model without those restrictions. It worked well enough that they pushed more than 17,000 recorded attacker events through LLM-driven analysis and reconstructed the full timeline in hours rather than days, with the additional benefit that none of the stolen credential data had to leave their own environment to be processed.
The asymmetry there is worth considering because the agent conducting the attack was operating under no usage policy whatsoever, while the team defending against it found their first-choice tooling refusing to engage with the evidence. Hugging Face was careful to say, correctly, that this isn’t an argument against safety measures on hosted models, and I’d say the same. It is a real operational gap, though, and their recommendation is sound: if you’re an organization that could plausibly be targeted this way, have a capable model vetted and ready to run on your own infrastructure ahead of an incident, because discovering mid-response that your tools won’t cooperate is an expensive way to learn this.
What This Means Going Forward
In immediate practical terms, there’s very little here for most readers to act on. If you happen to hold a Hugging Face account, rotate your access tokens and review your recent account activity, since that’s what the company recommended, though I’d guess that applies to a small portion of this audience.
The reason I wanted to write this up anyway is for what it establishes. Until last week, an autonomous AI system conducting a real multi-stage attack against real production infrastructure was something security people described as a forecast, and Hugging Face noted in their own disclosure that this incident matches the agentic attacker scenario the industry has been predicting. It now has a timeline, a write-up, and more than 17,000 logged events attached to it, which moves it out of the category of things that might happen.
What I’d take from it is less about this specific attack and more about cost. Multi-stage campaigns that previously required a skilled operator and a lot of patient hours are the sort of thing that gets meaningfully cheaper when software can run them, because software doesn’t tire, doesn’t bill hourly, and doesn’t lose interest halfway through. That doesn’t remove human attackers from the picture, it lowers the bar for the broad untargeted campaigns that used to filter out everyone except the genuinely committed.
Which connects back to something I’ve said here before: as finding and exploiting software vulnerabilities becomes more automated, the people running the software (as well as everyday people) become the more attractive target, because that’s where the automation helps attackers most and defends victims least. Expect more convincing phishing, better impersonation, and social engineering without the usual tells, since language models happen to be very good at exactly that. You’ll probably notice the effects of all this in your inbox before you notice them anywhere else.
An American lab’s models broke into a company because they were pointed at a benchmark and given no functioning restrictions, and a Chinese open-weight model helped clean it up because it was the tool that would actually engage with the evidence. Neither of those outcomes was anyone’s plan, he capability itself is indifferent, and what determines whether it secures a system or compromises one comes down to who’s holding it and what they’ve told it to do, which is apparently doing far more of the work here than anyone is comfortable admitting.
Read more: OpenAI | Hugging Face | TechCrunch | Scientific American