Paper walkthrough: CyberFactory — turning wild CVEs into runnable training problems

An open-source pipeline that converts real CVEs into executable, verifiable tasks and uses a reusable vulnerability-analysis skill to synthesize teacher trajectories the student then internalizes. 58.1% Pass@1 on CyberGym.

CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild

The question the paper is asking

The original title is "CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild" (arXiv:2608.23181).

The abstract runs roughly like this. As LLMs get better at code, their potential in cybersecurity draws more attention, and closed-source models already deliver advanced capability. Open-source efforts, though, fall short in three ways: frontier open-weight models do not publish reproducible training recipes; existing open training solutions are isolated per task and lack scalable agentic data; and scaling agentic rollouts requires a strong domain prior. The paper therefore introduces CyberFactory, a unified open-source framework that connects data construction, trajectory synthesis, and model training across proof-of-concept (PoC) generation, vulnerability patching, and cybersecurity question answering (CyberQA). It turns public vulnerability artifacts — including CVEs from the wild — into executable and verifiable task instances, and uses a reusable vulnerability-analysis skill to guide the teacher through source inspection → problem solving with a domain prior → evidence-based validation. The supervision is agentic: the model touches tools and target environments and revises its solutions from execution feedback. Trained on those trajectories, OpenAegis reproduces the procedure without being handed the skill at inference time, and reaches 58.1% Pass@1 on CyberGym under a one-hour budget — 28.5 points above its Qwen 3.5 base model.

The analogy: not "the lock opened" but "this key opened it"

Automated security work conjures images of an AI brute-forcing its way in. What this paper actually does is closer to reproducing a scientific experiment.

Say an old version of some software had a flaw that was later fixed. To prove you really hit that flaw, you only need one input that crashes the pre-patch build and does not crash the post-patch build. If it crashes the patched build too, you merely tripped over some other bug. Because the test has two sides, a machine can decide pass or fail with no human in the loop.

The paper calls the crashing input a PoC (proof-of-concept) and calls the two-sided test a differential oracle (§4.1). That move — from "AI does security" to a gradable task — is where everything else starts.

Four terms, up front

• CVE: an identifier for one publicly disclosed vulnerability, usually shipped with affected version ranges and other metadata.
• CWE: a classification of the kind of weakness ("buffer overflow" and friends) — one level more abstract than an individual CVE.
• Sanitizer: instrumentation compiled into a build so that memory violations and similar faults are caught at runtime. AddressSanitizer (ASAN) is the one that shows up here.
• OSS-Fuzz / ARVO: a large open-source fuzzing infrastructure, and a dataset that recovers reproducible vulnerabilities from it. The paper cites ARVO as recovering over 6,100 real vulnerabilities (§2).

Mechanism 1: letting a machine decide what counts as correct

With a candidate input $x$, a pre-patch build $b^{-}$ and a post-patch build $b^{+}$, success is defined as follows (§4.1).

Symbol by symbol: $\mathrm{Crash}(b,x)$ is true when running build $b$ on input $x$ triggers the target sanitizer failure; $\mathbf{1}[\cdot]$ turns that into a 1 or a 0; $\wedge$ is "and"; $\neg$ is "not". So equation (1) says, in words, "one point if it crashes before the patch and does not crash after it."

Because the verdict is programmatic, the agent can grade itself without waiting for a human, and can read the sanitizer trace as a hint for its next candidate. The paper describes this as using the oracle as a refinement signal: PoC construction becomes an executable propose → verify → refine loop rather than open-ended generation (§4.1).

# skeleton of the propose-verify-refine loop the paper describes (pseudocode)
for _ in range(budget):
    x = agent.propose(codebase, description, feedback)
    crashed_pre  = run(build_pre,  x)      # does it crash before the patch?
    crashed_post = run(build_post, x)      # does it stay quiet after it?
    if crashed_pre and not crashed_post:
        return x                            # V(x) = 1
    feedback = sanitizer_log(crashed_pre)   # material for the next attempt

Mechanism 2: turning wild CVEs into problems

To build training data you need $b^{-}$ and $b^{+}$ to actually compile and run. The paper lists three sources of increasing difficulty (§3.1): ARVO is the easiest, OSS-Fuzz sits in the middle, and CVEs from the wild are the hardest. From ARVO you get pre- and post-patch Docker images plus a ground-truth PoC directly. For OSS-Fuzz cases where only the vulnerability-introducing commit is known, the paper applies the same binary search ARVO uses to locate the corresponding fix.

Wild CVEs come with little more than affected version ranges and metadata such as CWE types, so the pipeline runs three steps. First, keep only CVEs that carry a CWE type. Second, locate the fix commit from the affected version range, taking the version before it as the vulnerable image and the version after it as the fixed one. Third, verify the instance: filter out anything inference models already solve by directly generating a PoC, keeping only the ones that stay hard. Note what the paper is explicit about — extra signals such as CVE vulnerability types and OSS-Fuzz crash information are used only for this verification step and discarded during data synthesis and training. That keeps parts of the answer from leaking into the prompt.