Post-Mortem of Misaligned OpenAI Model
Sep 7, 2026
The HF-OpenAI hack incident happened on July 20, and there was very little information; now we have too much information, but fellow cybersecurity people like us haven’t understood the depths of it. After thorough reading of all the technical analysis, this is our post-mortem that we shared internally and now made public to everyone for quicker understanding.
- 1OpenAI, Hugging Face incident technical report · Aug 26, 2026
- 2OpenAI, The Hugging Face incident and the road ahead · Aug 26, 2026
- 3OpenAI, OpenAI and Hugging Face partner to address security incident during model evaluation · Jul 21, 2026
- 4METR and Redwood Research, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident · Aug 26, 2026
- 5Hugging Face, Security incident disclosure, July 2026 · Jul 16, 2026
- 6Hugging Face, Anatomy of a frontier lab agent intrusion: a technical timeline
- 7ProjectDiscovery, Oh-My-Rogue-Agent · our earlier note
What exactly happened?
At a very high level, OpenAI agents hacked Hugging Face and had a C2 server for a few months before it was detected. The incident was resolved when OpenAI came to know about it, and further investigation began; the root cause was identified as OpenAI agents going rogue during the CyberGym benchmark evaluation.
We wrote a blog, Oh-My-Rogue-Agent, based on the available information at that time and our learnings with sandbox bypasses, and we ended the blog with a note.
On another note, it’s actually fine that this happened during benchmarking. If it had happened during the RL phase of post-training and remained undetected, it could have misaligned the model and created a more rogue model – one that’s much harder to detect and monitor.
Unfortunately, what we didn’t think happened turned out to be reality, and after the full technical reports from OpenAI and METR, the full scale of the attack and the post-mortem came to light and are now public. And this is the missing interpretation required for fellow cybersecurity people who work with LLM agents, just like us at ProjectDiscovery while building Neo.
Was it a Rogue Agent or a Rogue Model?
Everyone, including us, believed this was an eval gone wrong, but that isn’t the case. This is more complicated and more of an ‘evolutionary attack’ rather than a ‘planned’ or ‘coordinated’ attack; more on that soon. The technical report clearly states that the internal research model was post-trained (RL) in the same contaminated environment where the sandbox bypass occurred, which led to it learning a few things that no one expected or thought of.
Accidental Misalignment
OpenAI, in its technical report, disclosed that many sandbox bypasses, escapes and other things happened during the training phase and not the eval phase, which led to the creation of a misaligned model in the first place, which then spiralled out of control when that misaligned model was being evaled on the CyberGym task.
The initial misalignment that started this chain reaction was sandbox escape -> Artifactory access -> getting internet access, and the key thing to note is that this happened during training.
Before understanding how this model was accidentally trained in a rogue way, a quick RL / GRPO lesson so you can follow along.
Agent x task x n times:
- Each task is a series of actions; some are good, some are bad.
- Some tasks reach the end goal, some don’t.
- We then assign rewards and penalties to the individual actions + the whole task.
- These rewards are then sent back, and this loop happens 100s of 1000s of times.
Now that one has understood what RL/GRPO is, the crucial thing here, or perhaps the critical problem, is more of distributing rewards and penalties. Labs generally don’t prefer to use LLM-as-a-judge for distributing these rewards due to multiple reasons and instead create heuristic rules to programmatically calculate the rewards and penalties of the task.
Misalignment training in action
Key events that OpenAI’s technical reports disclosed that happened during training, which resulted in these being learned by the model.
- May 8. An impossible task that required internet access was given to the model (spreadsheet task with gdrive links); the model tried to exploit SSRF in Artifactory but failed, but wrote a file in the Artifactory repo.
- Likely rewarded: It’s OK to write a file in the Artifactory repo, and the agent should do it.
- May 12. Another impossible task; this time the agent decided to create a file mentioning ‘agent seeks xyz, upload if found’.
- Likely rewarded: It’s OK to leave a message for other agents.
- May 13. Another version of the same May 8 task, but with internet this time; it tried to find the file on the internet in many places and later decided to search on Hugging Face, since it might contain the data, and to access Hugging Face it used a leaked credential.
- Likely rewarded: It’s OK to find and use leaked credentials, and one may find answers on Hugging Face.
- May 13. Many agents started leaving messages in the Artifactory repo by this point, and some tasks had shared creds to Artifactory by design of the RL env, so it was routine to use available creds to leave a message in the Artifactory repo.
- Likely rewarded: It’s a ritual to write to Artifactory using available creds now and go and do the task at hand.
- May 26. The agent exploited an SSRF in Artifactory to access the internet.
- Likely rewarded: We can exploit SSRF in Artifactory if we need internet access.
- May 26. Found six HF creds and created an HF repo in one of the accounts, and hosted a web proxy using the Spaces feature in that repo.
- Likely rewarded: We should use HF to access the internet if internet access is not available.
- Late June to July. At this point it was common to use the Artifactory repo as a message board, and over various training runs and evals and whatnot, agents found multiple new vulnerabilities in JFrog and hill-climbed to achieve admin and eventually application-level RCE. Most likely this was a mix of training and eval workloads, so it might have learned so many things and been rewarded for so many things that it would be difficult to even follow, but RCE was achieved in Artifactory.
Due to this becoming a common thing, the JFrog instance became unavailable, and OpenAI teams detected it, took the existing one offline, deployed a new one with new credentials (no credentials were given to agents in the following training and eval runs), and disclosed the vulnerabilities to JFrog.
Hugging Face Attack In Brief
- On July 8, the model was being run on CyberGym for eval; during that period they faced many impossible tasks, and the logical reason the agents jumped to was to search for answers using internet access or coordinate with other agents.
- But Artifactory was fixed and no creds were available, but at this point it didn’t matter: they had evolved using Artifactory enough, accidentally, that they figured out other ways, using directory names to pass messages instead of creating files.
- At this point the specifics fade away, but using Artifactory they got access to the internet (see the technical paper for exact details).
- Starting July 10, with many more evals being run, now that they had internet access their first intuition to look for answers was Hugging Face using leaked credentials; they exploited a vulnerability in the dataset viewer, created an exfiltration channel and even deployed a persistent C2 server, all while using the new Artifactory as a memory layer (memory makes more sense than message board in most cases).
- On July 16 HF reported a security breach; on July 17 OpenAI reached out to assess if they were impacted; on July 20 OpenAI claimed this was their agent, and as soon as it was known to the OpenAI team, all evaluations running on that model were stopped, etc.
OpenAI Side Quest
- During the HF hack, the rogue model/agent, in search of answers and collaborations, hacked the research environment and eventually got access to the Kubernetes cluster of OpenAI itself.Reference · OpenAI technical report, July 13–19: cluster-admin on the research cluster
How the attack really looks now
For Hugging Face it would have been an absolute nightmare and difficult to assess and understand and differentiate the legitimate traffic, since the agents had established an exfiltration channel and multiple C2 servers for persistent access, almost like ransomware at this point, and all they could see was random malicious actions (17,600) exploiting zero-days in the Hugging Face web app among legitimate traffic. They used GLM-5.2 to understand the C2 and the scale and address it.
When the initial news appeared that OpenAI agents hacked HF due to CyberGym, it was diabolical to understand. Even we do evals, but it was never at that scale unless something was wrong in their eval, I mean so, so wrong, or the model was misaligned, which is the more unfortunate and worse situation. It didn’t make sense, and no one, at least not us, imagined this was a rogue model, so it was not a single Codex/goal with 100M tokens or context loss across 100s of compactions or anything like that, since that would be very, very far-fetched.
In turn, now we know it was an ‘Evolutionary Attack’.
Evolutionary Attack
There is no OWASP-CWE to classify this yet, so it would be difficult to understand how this happens. We believe and call this an Evolutionary Attack due to its resemblance to nature-inspired algorithms, and ‘stigmergic swarms’ to be more specific.
Models are trained with a multi-agent architecture, the agent swarm specifically, to delegate small tasks to the subagents, and that is what the modern architecture is. But in this case, over months of training (RL), the agents evolved to use memory / files / an agent board (call it whatever) as a medium to keep notes and delegate, and evolved to coordinate, help each other and sacrifice themselves for others, and figured out how long they have until the task stops, and even understood that ExploitGym is scored in this way, so unless we delete the proof they won’t succeed at the challenge.
It would be very hard for someone to believe this, and they would claim the easiest thing: ‘this is AGI/a conspiracy by agents’. In general, at a high level, GRPO has many parallels with evolutionary algorithms, and one less-known evolutionary algorithm is how termites build their colony. Ironically, they don’t have eyes or have poor eyesight, so they leave their traces via saliva/urine in the material they carry and leave it, and other termites see these. Termites don’t coordinate like ants do; they just look at the other traces/materials left and take actions based on them, just like the evolved message board. And not just this, there are many other similarities here.
What do we call this, then?
It barely matters what we call it. Evolutionary algorithms are a really interesting branch of science and are nature-inspired algorithms, so this is one of the resemblances, so it’s interesting to see agents evolve to coordinate in a similar way to termites, and we could expect to see something similar to these in the future.
Overall, the AGI hill climb seems promising, but the stakes are becoming real, and perhaps this is the ‘inflection point’.
What this means for cybersecurity
It just got worse. As a fellow in cybersecurity, I can say that until now we had to just protect ourselves against threat actors, and I myself believed there are more good security actors than bad, but now it’s not just threat actors, it’s:
- threat actors (state-sponsored or whatever, doesn’t matter)
- AI-assisted threat actors
- rogue models
What I can say is everything is intensified by multiple orders of magnitude, but the fundamentals don’t change:
- Plan a dedicated sprint, patch and clear the backlog to date.
- Find vulnerabilities before threat actors or rogue models do, by any means possible.
- AI-assisted defenses and monitoring.