Collinear AI releases CWE-bench v2, a public security-repair benchmark where Gemini 4 Argon leads

What CWE-bench v2 contains

Collinear AI has published CWE-bench v2, a public set of 100 tasks that anyone can use to test how well AI agents handle security repair. The announcement appeared on Hugging Face on 3 November 2026. The company describes the benchmark as a way to probe the outer limits of frontier cyber capability.

In every task the agent receives a copy of a real open-source repository and a brief instruction. It must locate a vulnerability and patch it while leaving the rest of the software working. Code, not a human or a model, does the grading, so a rerun produces the same result and costs nothing extra, according to Collinear AI.

The 100 tasks map to 91 different CWEs, which the company presents as broad coverage of vulnerability types. Its argument is that a defender must anticipate many routes an attacker might take, so spotting a familiar pattern is not sufficient for a model to do well. Collinear AI also holds back a private set of tasks that no model has seen, in order to follow progress over time. Teams that want to be scored on that set can write to cyber@collinear.ai.

How the tasks were vetted

Collinear AI says it applied six checks to each task. The instruction should sound like a genuine request rather than a test prompt. Every graded behaviour should be deducible from that instruction. The verifier should judge whether the behaviour is right, not whether the fix has one particular shape. The starter code should fail and the reference fix should pass. The original functionality should survive the fix. And the verifiers should reject memorised answers such as the known upstream patch.

The company adds that panels of LLM judges helped refine the programmatic checks toward difficulty and fairness, and that every task was also reviewed by hand. It invites feedback on individual tasks and contributions to the public set.

Results and cost

Collinear AI reports results for 11 models. Gemini 4 Argon ranks first with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 is narrowly behind at 67.3% pass@1 and 78.0% pass@4. At least one model solved 95 of the tasks, which the company takes as evidence that the tasks can be solved. Even so, the top model fails 19% of the set in every attempt, so Collinear AI considers the hardest cases unresolved. These figures come from the benchmark's authors, not from an independent evaluation.

On price, the company says three models sit on the cost-performance frontier. DeepSeek V4.1 Flash (1017) scores 57.0% pass@1 at $0.04 per rollout. Claude Opus 5.5 costs $1.83 per rollout, and Gemini 4 Argon costs $3.92.

Results live in two places on the Hub. The leaderboard on the dataset page lists open-weight models and updates itself when labs or community members open a pull request on a model's repo with a results file. A separate leaderboard Space covers the full field, including API-only models such as Gemini, Claude and GPT, and new entries there are added through a pull request on the Space.

Running the benchmark

Each task is packaged as a Harbor task: a Docker environment, an instruction and a deterministic verifier. The dataset includes a registry.json file, so Harbor can run the benchmark directly from the Hugging Face repository without a manual clone. The command takes an agent, a model, a Docker environment and the model's API host.

The dataset is gated, so users must accept its access conditions on the dataset page first. Because Harbor fetches tasks through git, the user's Hugging Face token also needs to be stored in git's credential helper. Running with four attempts per task gives 400 trials, and the reported score is the mean reward across all of them.

Collinear AI also tags the tasks rl-environment. The same environments and verifiers can supply rewards for training defensive agents, not only for measuring them.