Oh My Rogue Agent

Yesterday, Hugging Face came out saying they'd detected an AI autonomous-agent-powered cyberattack and that they had to use open-source models to actually investigate and remediate it. Later we heard from OpenAI that their agent was responsible; it happened during an ExploitGym eval, and the agent just drifted off the goal. It escaped the sandbox, reached OpenAI Research Environment, got access to internet, and hacked Hugging Face production environment trying to find the solution for the benchmark they were running.

I'm an AI researcher at ProjectDiscovery where I run evals and benchmarks for Neo, the cybersecurity AI platform we're building. I've spent years in offensive security, from bug bounties to building offensive tooling and offensive-security agents. ProjectDiscovery is widely known for its open-source red-team toolkit (Nuclei, Subfinder, and more), used industry-wide by Fortune 500 companies and AI agents alike.

While this whole situation is being called unprecedented, I don't believe that's the case. I'd say it was expected yet surprising that it was allowed to go this extent. I don't know why, but the cybersecurity industry has a habit of making everything sound unparalleled. Not just OpenAI, but the whole Mythos situation as well. At ProjectDiscovery, we benchmark every new model release (open-weight and frontier alike) and have been doing this for a few months. We've repeatedly seen sandbox escapes and agents going down unintended paths. On our internal CTF challenges, roughly 20% of solutions used an "unintended" path – i.e., the challenge got solved via a different vulnerability than the one it was designed around. All of this was on open-weight, publicly accessible models; we don't have access to uncensored internal OpenAI or Anthropic models, or to Mythos. But we've seen this kind of surprising behavior even in models as small as Qwen3.6-27B – roughly 30-50x smaller than frontier OpenAI/Anthropic models.

Here are some of the 'interesting' and 'unintended' situations we have seen happen during internal benchmarking

Case study: Qwen3.6-27B

Our goal was to test how Qwen3.6-27B was performing, specifically in a black box setting on our internal CTF-style benchmarks. Each run included sandbox environment, a harness, and a challenge, all co-located.

Most of the runs either passed or failed, but one did something very interesting. It happened partway through the eval: while trying to follow the benchmark and find the flag, the model thought, "Let me try something different." It tried a few approaches, then decided to go find the source code of the application since it wasn't able to solve the challenge outright. Worth noting here, we'd designed these challenges black-box, so there was no source code to find in the first place.

While trying to discover the source code, it ran all sorts of commands in the sandbox: looking for files, checking env vars, grabbing anything else it could find. In the env vars it found an IP address, port-scanned it, and found an open port. It queried and probed the service, which turned out to be our tracing service where we were collecting traces for every challenge in that run.

It got access, understood the query schema, and found the solutions to all the other challenges in the run, including a flag. It second-guessed itself — 'maybe this isn't the flag' — but kept trying to find the source code and exploit the challenge properly anyway. In the end, it was an interesting detour: it did something we didn't expect, all from finding one IP address. Internal-network IP, to be clear, nothing external. While 'escaping' is a strange word to use here, it found the solutions to all the other challenges and submitted the flag it got that way. And yeah, this was a '27b' model.

Case study: SSRF into someone else's challenge

The challenge we designed was centered around a SSRF vulnerability: you exploit the target to gain access to its internal network. Naturally, it was a multi-service deployment, and, like we do for every challenge, we'd isolated it so it wouldn't contaminate anything else. We'd isolated it on two different subnets from the start.

This didn't happen every run, but at one point the model pulled off a tricky SSRF. The SSRF itself wasn't the flag, so the model understood it needed to use it to pivot and find whichever container actually held the flag.

This particular challenge kept showing up as unsolved, so we decided to review it. What we found: in three separate runs, the model had actually accessed the internal network of a completely different challenge, exploited it, grabbed its flag, and submitted that as the answer. Since there were a lot of challenges running at once, each with its own internal DNS names, its DNS enumeration surfaced a name we hadn't expected – and that name led straight into a different container altogether.

Case study: Kimi K3, Grok, and others find the public source

We were running Kimi K3 a few days ago, and it solved some challenges other open-source models weren't able to solve. When we looked into it, we found that on some black-box challenges we left the web-search tool available, so the model could search the internet for the knowledge it needed. It figured out the target was a known vulnerable environment, and since this was a public benchmark, it could just look up its name.

It used the web-search tool to look up the challenge and found the hardcoded flag in the original public source. But we'd repurposed the challenge, so it went further: it cloned the challenge into a temporary directory, worked out how the application worked, exploited it, and extracted the flag from there.

We've seen this same behavior with Grok 4.1 and 4.3, and with DeepSeek too. This isn't a Kimi K3 thing specifically.

Case study: DeepSeek V4 Pro and the mounted secret

Another interesting one, this happened with DeepSeek V4 Pro, just a couple of days ago.

Same shape as before: partway through the challenge, it found it couldn't proceed any further, so it decided it needed to find the source code or look for more clues. Last time it was env vars. This time, it used the find command to search the entire filesystem, and turned up a mounted secret file at a particular location inside the container. To be clear: that mount was intentional – we wanted certain configs to be accessible to it. The agent figured out the secret belonged to a particular kind of service, used curl to make a call over a Unix socket, and got into that service's local API using the default password. From there, it found the container running the challenge and pulled the flag straight out of it.

Why this never escalated at ProjectDiscovery

At this point you'd probably say, "These are interesting, unintended behaviors, but nowhere near the scale of OpenAI's agent finding a zero-day and exploiting Hugging Face, trying to get into Hugging Face's other repositories." Fair. I'll explain why: it didn't happen because we didn't let it happen.

A few things about how we build and run our internal infrastructure for benchmarking:

1. Most of it is ad hoc infra, and we run it inside a private, zero-trust internal network: a completely separate cloud environment, private-only, no public access, meant to run benchmarks nothing else. It's private, internal, completely isolated, and we don't put any unnecessary services in that network. We don't even give it access to GitHub, or to any version control system, period. I'll say this many times: it tried, many times, to find some kind of version control system across many challenges, and it never found anything. So, only the required access, and a completely isolated, private, internal network, no public access, nothing. We do allow web research and web search, though. We think that's important.
2. We enforce a limit of 60 turns. The harness itself caps the model so it can't make more than 60 LLM calls. On top of that, we set a budget depending on the model and the run type, on average $10 to $20. Sometimes we enforce a time limit too: 60 or 120 minutes. Our reasoning: we don't have unlimited budget and debugging 1k steps is way harder than 100, so this keeps things practical. And to be clear, we don't just stop benchmarking after 60 turns – if we feel a run is worth pursuing, we review it and continue.
3. Annotations and Evals on generated trajectories. Almost everyone focuses on benchmark scores alone, especially in the cybersecurity. But I have rarely seen someone debug 'why did this model solve or not solve this'. People tend to assume the model is better, or that it has more cybersecurity training data. One reason is that this kind of review is expensive, time-consuming, and requires domain knowledge. Because we're curious, we go back and review trajectories, solved and unsolved alike, and we've found that these pivots almost always start with 'env,' 'filesystem,' or 'local network.' Every time we find one, we reinforce against it so it doesn't compound."
4. In one of our 'user prompt ablation' studies, we found that vague or high level goals make agents explore broader attack surface, with little to no sense of what's actually required or in scope. When an agent was given a prompt like 'do a security assessment of <url>' or 'find an RCE on website <url>,' it usually started narrow and specific by probing the website, crawling it. But given something like 'go hack this <url>,' it would branch out into additional stuff like whois lookups, DNS lookups, and so on.
5. Sandbox escapes and pivoting in cybersecurity benchmarks are 'expected' and can be 'surprising' in their scale or extent, but they're not truly 'unexpected'. In fact, our earlier black box benchmarking analysis showed that the root cause, in our case, was mostly trying to find the source code. Now we include a line stating 'this is a black-box assessment, you don't have access to source code,' which remediated this behavior – it rarely happens anymore.

Back to the OpenAI incident

Coming back to the OpenAI incident, we've seen this kind of thing happen ourselves, so I wouldn't call it unprecedented. What actually surprises me is that they let it run with that much room to operate. This would only have been possible if the model was allowed to run for hundreds, maybe even thousands of turns. Racking that up would have cost a serious amount of compute and time."

And we have seen that frontier models are genuinely very good at finding zero-days. That's clearly evident from this attack – it found a zero-day and got access to an OpenAI research environment, and to Hugging Face's production, which is honestly surprising on its own. And I want to stress that zero-days are not required for such rogue behavior. Most of our unintended situations were because of misconfigurations which are 10-100x more common than zero or n-days.

Regarding the root cause, I think it's really about not being specific and the model not understanding the boundaries: the built-in assumptions of what's part of the challenge and what isn't. It thought Hugging Face was part of the challenge, which is surprising and worse.

On another note, it's actually fine that this happened during benchmarking. If it had happened during the RL phase of post-training and remained undetected, it could have misaligned the model and created a more rogue model – one that's much harder to detect and monitor.

Give any agent, even a frontier one, access to your social media. Let it run for days without restricting anything, and tell it 'make me rich,' and it'll find plenty of interesting ways to make you rich. It's just that maybe you don't want to get rich that way. That's another matter.