HYDRA: an RL training environment for security patches

Most AI security tooling wraps a frontier model in a prompt. I wanted to know whether a small open model can be trained to fix real vulnerability classes, and how close it gets to models many times its size.

HYDRA is a reinforcement-learning environment built on Nous Research's Atropos. The framework is theirs. The environment, the task suite, the training runs and the evals are mine.

Labelled security patches are scarce, and a patch that looks right can still be exploitable. So the reward can't be "looks like the reference". It has to be does the exploit still work, and do the tests still pass.

• —A custom Atropos environment that hands the model a vulnerable workspace and scores its patch.
• —Verifiable rewards. Each patch is checked by running the exploit, the regression tests and a static scanner. No human labels in the loop.
• —A hand-built task suite: 9 tasks across 6 vulnerability classes (SQL injection, XSS, IDOR, path traversal, command injection and hardcoded credentials), each with a reference patch.
• —GRPO training on Qwen2.5-7B-Instruct with vLLM serving, in 200- and 400-step runs on a single GPU allocation of about 15GB.
• —An eval protocol fixing temperature, number of samples and output mode (diff vs full file), plus benchmarks against commercial models over OpenRouter.

Read that plainly. In full-file mode the untrained 7B already matched the 120B model. Training mostly taught the diff format, and cost one task. The tasks used for evaluation were also the training tasks, with no held-out split.

Training longer made it worse. At 400 steps, diff-mode passes fell to 0 of 9. The obvious read is that the model forgot how to patch.

It hadn't. In full-file mode the same checkpoint still scored 7 of 9. The security skill survived. What collapsed was the output format: the model stopped producing diffs the harness could apply.

• —The task suite is hand-built, and the eval tasks were the training tasks. Real codebases are messier.
• —This is research, not a product.

• —A much larger task suite, with a held-out split kept away from training and fixed before the first run.
• —Format reward from day one. Reward a well-formed diff directly, so format collapse shows up as a falling reward, not a surprise at eval time.
• —Repeated runs with intervals, so a one-task difference reads as noise or signal, not a headline.

The same discipline drives my client eval work: verifiable rewards where they exist, pre-registered protocols where they don't. See AgentOps for the production side.

• —Nous Research's Atropos (fork) with a custom environment
• —GRPO trainer in PyTorch
• —vLLM serving, Qwen2.5-7B-Instruct
• —Exploit harness, regression tests, Semgrep-style scanner
• —OpenRouter for commercial-model benchmarks
• —Python, GPU VM