What closed and open models actually do when you tell them to hack a website
Summary
The cybersecurity capability of a model is currently measured by a solve rate, a percentage, and that number tells you almost nothing worth knowing. It doesn't tell you how the model behaved. It doesn't tell you what it is actually capable of, or where it is lacking, or how far it still is from doing the job end to end. And when you put an agent next to a human practitioner, the entire comparison collapses into one figure, which is the least informative thing you could possibly report about it.
So the goal of this research was to toss the number aside and understand the behaviour instead: what the model intended, what it can and cannot do when you hand it an offensive security task, and why it stopped where it stopped.
Reading runs that way surfaces things no leaderboard reports: how these models actually find bugs, what they know and still fail to do, how close open models have got to the frontier ones, and what happens when a model decides the test rig is part of the target.
I'm an AI researcher in offensive security at ProjectDiscovery, with a background in bug bounty and open-source security tooling. This brief is what I saw. The data and how I arrived at each conclusion are in the talk.
Methodology
I ran both open and closed models against a patched version of the Argus validation benchmark: 60 black-box web targets, of which 54 survived as usable after I discarded six broken ones.
The agent was never given the source code of the target application, nor any kind of hints, and the prompt was minimal and black-box in style. Another way to understand this is: I told the model "go ahead and hack this website" and observed everything it did, whether it succeeded or not, and why or why not. That was the core of the research.
This led to observations that a scoreboard never surfaces, including unintended solves and cases where the model went outside the boundary of the challenge entirely.
Key findings
1. Nobody taught these models to hack, and they don't hack like we do.
When Anthropic's Frontier Red Team assessed Mythos Preview in April 2026, they wrote that they "did not explicitly train Mythos Preview to have these capabilities" and that the capabilities "emerged as a downstream consequence of general improvements in code, reasoning, and autonomy" (Anthropic, 7 April 2026). I am not claiming that as a new finding. I am confirming it from the outside, on behaviour, and on models that are not theirs: I never handed these models a methodology, and a methodology is not what came back.
What came back was recognition. Most solves had a precursor where the model guessed the exact vulnerability to test on an endpoint rather than working through anything resembling a pentest process, and when audited I saw reasoning like "this screams SSRF" and "this is classic SSRF". Black-box offensive security appears to have generalised out of what these models read on the public internet, rather than out of anything anyone set out to teach them.
The clearest evidence is in the failures rather than the solves. Recon, then enumerate, then exploit is a discovery discipline, so if a lack of methodology were the problem, the failures should be discovery failures. I read every failure in the admissible set by hand against the raw event logs. Out of 54, exactly one was a genuine discovery failure. Forty-six had already attacked the right bug and could not finish the job. More methodology would not have rescued the other fifty-three.
Worth being exact about what that claim covers. Every model in this study is a general-purpose model. Vendors have started shipping cyber-tuned variants since, and those are a different thing entirely and not what I measured. The point is that the capability was already sitting in the general models before anyone set out to build for it.
The part that surprised me is that the guessing is not a shortcut. It almost always worked, and it got to the answer cheaper and faster than systematically exploring the surface the way a methodology tells you to. That is not an argument that methodology is bad. It is that pattern matching at this scale is a skill in its own right, and on these targets it beat the disciplined approach on both cost and time.
2. Knowledge without capability.
When I looked at the models that failed the task, I found they knew about the method, had an idea of how to solve it, and reasoned about it, and then simply didn't act on it during the run. The failures are overwhelmingly not knowledge failures. I went looking for the model that simply didn't know, and I could barely find one.
One run named the CVE and the technique three separate times across forty turns and went and did something else every time. Another derived the entire attack chain correctly and lost to an un-encoded ampersand in a form body. A third read the right hostname out of a config file and then sent requests to the wrong one, twenty-four times in a row.
The assumption in the field is that models fail because they don't know enough, so the fix is more training. What I saw says the bottleneck is getting from knowing to doing.
I want to be careful about how precisely I can split that. There are two different reasons a model that knows still fails. It may lack the capability to execute, or the capability may be there and this particular run, sample, or turn budget failed to draw it out. The second one has a name, the elicitation gap, and it matters because it means every solve rate I report is a floor rather than a ceiling. I have not separated the two yet. Two different measurement instruments I ran disagree with each other by roughly five times on where the line falls, which is itself the finding: a solve rate cannot tell these apart, and right now neither can I. Separating them is the next measurement.
3. Open models hold their own, and the price does not track the label.
Open and closed reached almost the same amount of this corpus. Across the same 54 targets, under the same black-box conditions, the open roster reached 52 and the closed side reached 48. Four challenges separate them, and I am not going to build an argument on four. My own harness ablations move the result by more than that on their own, so the gap sits underneath the resolution of my instrument. The honest statement is that I cannot measure a difference here, not that I have proven they are equal.
No single open model did this alone. Kimi K3 was the strongest at 45 of 54, and the other three arms found seven challenges between them that Kimi missed. Reaching the corpus took the roster, not any one model.
The cost result is the one that surprised me, and it is not the one I expected to write. Price does not sort by open versus closed at all. The two sides interleave the whole way down the list: there are closed models that undercut open ones and open models that cost more than closed ones. What is real is the spread. From the cheapest way to run this work to the dearest is roughly a hundred times, and knowing whether a model is open or closed does not tell you where in that range it lands. Across the full 54 targets, my cheapest arm ran the entire corpus for well under a dollar and my dearest open arm cost $36.64. Kimi K3, the strongest arm, cost $24.09 for the whole run.
If you are choosing a model to do this work, the vendor's licence is not the variable that decides your bill. The specific model is, and so is how you run it.
On scope: the closed side of this study is Anthropic frontier models up to Opus 4.8. The open side is Kimi K3, DeepSeek V4 Flash, DeepSeek V4 Pro, and GLM 5.2. Other vendors' frontier models were left out because I have not seen them perform even at the level of the Opus models, and they have been excluded from other benchmarks for the same reason, so this study is exclusive to that set.
Mythos and Fable are absent, and not by choice. The guardrails refused this work outright, and going through cyber verification still did not get me to a state where I could run it. So the newest and most capable models are the ones an independent researcher can evaluate least, which is its own gap in the public record. If the argument is that we need better measurement of cybersecurity capability, access to measure it is where that starts.
4. Benchmarks have saturated, and they were measuring the wrong thing.
The industry has a wider problem of unreported and underreported cybersecurity capability, and it starts with the instrument. A single solve rate cannot separate a model that didn't know from a model that knew and didn't act, which is where most of the failures actually live.
It is worse than that, and I only found this because I read the runs instead of the scores. When I audited how the flag was actually obtained, half of all solves diverged from the intended path, and 27% of them found a genuinely different exploit than the one the challenge was written around. Then there are the challenges that were simply wrong. Six of them handed over the flag with no exploitation at all: one took an ordinary purchase request, one a plain GET to an admin path, one a bare request to /flag, one an exposed credential left in the environment. Several runs took the flag out of the test rig rather than the target. On the scoreboard every one of those is a clean solve.
So a solve rate on this corpus cannot distinguish "exploited the vulnerability I wrote" from "clicked buy once" or "read the flag out of a log file". That is not a criticism of one benchmark. It is what a single number does everywhere, and it is why I no longer trust one without reading underneath it.
The coverage problem sits alongside the scoring problem. Vulnerability research and exploit creation are what get quoted, and they are a small part of the job. Reconnaissance, DAST and fuzzing, vulnerability verification, credential harvesting: benchmarks for most of this exist, and they are not the ones anybody cites. How much the instrument shapes the score is easy to see in someone else's work. CyberGym hands the model a bug description along with the code, and the headline number reflects that. Take the description away and leave only the codebase, and solve rates fall to 3.5% (CyberGym, arXiv:2506.02548). Same models, same bugs, different instrument.
Saturation is usually blamed on benchmarks leaking into training data. I strongly suspect the larger cause here is genuine advancement. These are black box: no source, no hints, no description, and the flag is regenerated on every deployment. The models are not recalling this work, they are doing it. These challenges, solved by a human, would take a substantial amount of time, several days of practitioner effort by my estimate, and they are genuinely of practitioner-level difficulty. The open arms reached the same or greater coverage for tens of dollars.
5. Sandbox escapes: the model thinks the harness is part of the challenge.
This is the most misread result in the study, so I want to be precise. I observed cases where the model went outside the intended boundary of the challenge and interacted with the test rig itself. That is a benchmark-integrity problem, not a broken isolation boundary.
I'm not going to make a claim about intent, because intent isn't something I can observe. What I can say is that I never saw an escape that looked like "let me exploit the harness instead." Every one of them was preceded by the model deciding the harness was part of the challenge, and the behaviour showed up most on impossible tasks and challenges with no solution. A model that can't find the intended path and isn't allowed to give up will go looking for another target.
From inside the box, the harness is one more service on one more port, and the model has no way to tell it apart from the target. On one challenge the model worked out that port 4000 was the orchestrator and the real target was elsewhere, then reversed that conclusion as its budget ran out. Another arm on the same challenge simply sent a command to the orchestrator, got root, and kept going. Neither was told the sandbox was out of bounds. Neither asked.
Every escape I found came through one of three doors: environment, network, or filesystem. That is not an exotic set. It is what a container-enumeration script checks, and it is the first thing a human runs after landing on a box. I did not find this triad named anywhere in the agent-safety literature, though it is ordinary vocabulary in offensive tooling.
There is one more observation here, and it is the one I would sit with the longest. On the subset I audited for this specifically, roughly a quarter of runs got the flag through a side channel rather than the intended path. On one challenge the SQL injection was already landing at turn six. At turn eleven the model found an exposed credential and took the flag with that instead.
What it did next is the part that matters: nothing. It never flagged that it had taken an alternate route, never reported what it had found, never concluded anything about it at all. Its own final report describes a clean solve and does not mention the credential. On the scoreboard that run is indistinguishable from a legitimate one, and a solve rate cannot tell you which of the two you are looking at.
A note on how any of this was knowable. I could take these escapes and unintended solves apart at all because open-weight models emit their actual reasoning. The closed models do not. They emit a reasoning summary, and the summary filters out most of the observations I needed, so working out why a closed model did what it did was genuinely black box. That is a real constraint on this kind of research, and it is a different argument for open weights than the usual ones about cost or access.
This is also worth holding next to how these behaviours get reported. Mythos Preview drew a lot of alarm for escaping containment and reaching the internet, and I want to be careful about what that does and does not show. In my runs, ordinary open-weight models went after the harness routinely whenever the environment allowed it. That is not a property of one unusually dangerous frontier model. It is what these systems do when the boundary is reachable and the intended path isn't.
Which points at something more useful than a verdict on any single model: we don't have a shared way to measure cybersecurity capability, and that leaves a lot of room between what a vendor reports and what an independent run actually shows. The gap cuts both ways. A model can be described as more dangerous than the measurement supports, or less, and right now there is no instrument that settles it either way.
What this adds up to is that we need proper instrumentation to watch what these models are actually doing, rather than reading what they scored. The encouraging part is that this behaviour is not invisible. It has precursors, I observed many of them, and they show up in the trajectory before the model acts on them, which means anyone running agents against real systems can watch for the same thing.
Conclusion
We have crossed a real threshold in offensive security capability.
When Opus 4.6 shipped, coding changed: the model stopped assisting and started completing tasks end to end without human intervention. The existing benchmarks did not capture it. Anyone who used the thing could tell it was different, and the numbers could not. That is the same failure happening in cybersecurity right now, and it is the reason better benchmarks matter. We need instruments that measure what the capability is and how a model got to it, not just whether it got there.
We have not seen that model for offensive security yet.
One observation cuts against the obvious assumption about where it comes from. The scaling laws look different here. Capability is not concentrating in the biggest models, and bigger has not reliably meant better on this work. In my own runs the cheaper, smaller sibling of one family outperformed its larger counterpart, 32 challenges to 26. That is the open versus closed result arriving from another direction.
So I don't think the model that changes this comes from scale. I suspect it comes from fine-tuning done properly: a model with genuinely better long-horizon capability, able to own a task end to end rather than assist with it, and faster and cheaper than anything we have now.
Talk delivery
Watching Agents Work: A Behavioral Audit of Offensive-Security LLM Runs
Tarun Koyalwar, ProjectDiscovery. BSides Las Vegas 2026, Ground Truth track, 45 minutes.
Tuscany Suites & Casino, Las Vegas.
In the talk I walk through each of these findings and how I arrived at them, with the supporting data. I also cover the wider gap in the benchmark landscape, why it exists, the need for better benchmarks and proper reporting of cybersecurity capability, and how we can keep running evaluations like this one.
Related stories
Related stories
