--- title: "Collinear AI releases CWE-bench v2, a public security-repair benchmark where Gemini 4 Argon leads" url: https://llmreport.gr/news/collinear-ai-releases-cwe-bench-v2-a-public-security-repair-benchmark-where/ language: en published: 2026-11-04T01:11:25.860Z updated: 2026-11-04T01:11:25.865Z site: "LLM Report" section: "Models" --- # Collinear AI releases CWE-bench v2, a public security-repair benchmark where Gemini 4 Argon leads Collinear AI opened 100 tasks from CWE-bench v2 to the public on 3 November 2026. In its own tests of 11 models, Gemini 4 Argon scored highest, with Claude Opus 5.5 close behind. > Written with AI from the sources listed below. Reviewed by Konstantinos Perros. How we write: https://llmreport.gr/how-we-write/ ## Key facts - CWE-bench v2 has 100 public tasks spanning 91 weakness classes, and Collinear AI also keeps a private held-out set. [1] - Each task asks an agent to find and fix a security flaw in a real open-source repository without breaking existing behaviour, with deterministic programmatic scoring. [1] - Of 11 evaluated models, Gemini 4 Argon scored 67.8% pass@1 and 81.0% pass@4, and Claude Opus 5.5 scored 67.3% and 78.0%, according to Collinear AI. [1] - DeepSeek V4.1 Flash (1017) reached 57.0% pass@1 at $0.04 per rollout, against $1.83 for Claude Opus 5.5 and $3.92 for Gemini 4 Argon. [1] ## What CWE-bench v2 contains Collinear AI has published CWE-bench v2, a public set of 100 tasks that anyone can use to test how well AI agents handle security repair. The announcement appeared on Hugging Face on 3 November 2026. The company describes the benchmark as a way to probe the outer limits of frontier cyber capability. In every task the agent receives a copy of a real open-source repository and a brief instruction. It must locate a vulnerability and patch it while leaving the rest of the software working. Code, not a human or a model, does the grading, so a rerun produces the same result and costs nothing extra, according to Collinear AI. The 100 tasks map to 91 different CWEs, which the company presents as broad coverage of vulnerability types. Its argument is that a defender must anticipate many routes an attacker might take, so spotting a familiar pattern is not sufficient for a model to do well. Collinear AI also holds back a private set of tasks that no model has seen, in order to follow progress over time. Teams that want to be scored on that set can write to cyber@collinear.ai. ## How the tasks were vetted Collinear AI says it applied six checks to each task. The instruction should sound like a genuine request rather than a test prompt. Every graded behaviour should be deducible from that instruction. The verifier should judge whether the behaviour is right, not whether the fix has one particular shape. The starter code should fail and the reference fix should pass. The original functionality should survive the fix. And the verifiers should reject memorised answers such as the known upstream patch. The company adds that panels of LLM judges helped refine the programmatic checks toward difficulty and fairness, and that every task was also reviewed by hand. It invites feedback on individual tasks and contributions to the public set. ## Results and cost Collinear AI reports results for 11 models. Gemini 4 Argon ranks first with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 is narrowly behind at 67.3% pass@1 and 78.0% pass@4. At least one model solved 95 of the tasks, which the company takes as evidence that the tasks can be solved. Even so, the top model fails 19% of the set in every attempt, so Collinear AI considers the hardest cases unresolved. These figures come from the benchmark's authors, not from an independent evaluation. On price, the company says three models sit on the cost-performance frontier. DeepSeek V4.1 Flash (1017) scores 57.0% pass@1 at $0.04 per rollout. Claude Opus 5.5 costs $1.83 per rollout, and Gemini 4 Argon costs $3.92. Results live in two places on the Hub. The leaderboard on the dataset page lists open-weight models and updates itself when labs or community members open a pull request on a model's repo with a results file. A separate leaderboard Space covers the full field, including API-only models such as Gemini, Claude and GPT, and new entries there are added through a pull request on the Space. ## Running the benchmark Each task is packaged as a Harbor task: a Docker environment, an instruction and a deterministic verifier. The dataset includes a registry.json file, so Harbor can run the benchmark directly from the Hugging Face repository without a manual clone. The command takes an agent, a model, a Docker environment and the model's API host. The dataset is gated, so users must accept its access conditions on the dataset page first. Because Harbor fetches tasks through git, the user's Hugging Face token also needs to be stored in git's credential helper. Running with four attempts per task gives 400 trials, and the reported score is the mean reward across all of them. Collinear AI also tags the tasks rl-environment. The same environments and verifiers can supply rewards for training defensive agents, not only for measuring them. ## Questions and answers **What is CWE-bench v2?** It is a public set of 100 tasks from Collinear AI in which an agent must find and fix a security weakness in a real open-source repository without breaking existing functionality. The tasks cover 91 different CWEs. **Which model scored highest on CWE-bench v2?** Among the 11 models Collinear AI evaluated, Gemini 4 Argon scored highest, with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 followed with 67.3% and 78.0%. **How can I evaluate my own model on it?** Accept the access conditions of the gated dataset on Hugging Face, save your token to git's credential helper, and run the benchmark with Harbor. Results for open-weight models can be added by pull request, and API-only models are added through the leaderboard Space. For the held-out set, contact cyber@collinear.ai. ## Sources 1. Hugging Face: [CWE-bench v2: More coverage, quality, and transparency](https://huggingface.co/blog/cwe-bench)