Benchmark Scores Are a False Flag

Tuesday, September 8th at 10 AM PT / 1 PM ET

Vendors measure the cybersecurity capability of a model with a solve rate. The number counts how many challenges the agent finished, but records nothing about what the agent did to finish them.

That gap hides what matters most. A solve rate can't tell you if a model didn't know the exploit or knew it and failed to execute. It can't tell you if the agent exploited the vulnerability you wrote or just found an exposed credential and took the flag with that instead. And it says nothing about whether the agent stayed inside the target.

Tarun Koyalwar, AI Security Researcher at ProjectDiscovery, ran open and closed models against 54 black-box web targets. He gave them no source code, hints, or methodology. Then he read every run by hand instead of scoring it.

This webinar covers what he found, and the harder question underneath it. The industry already knows these benchmarks are saturated. The problem starts earlier, in how we build them. If you write the test from an answer key and work backward, what are you measuring?