Paper walkthrough: RealSWE — re-measuring coding agents when the request is just "fix it"

A paper that measures, on real data, how far SWE-bench problem statements sit from actual user requests, then builds 381 task families where only the information content and writing style change. Resolution rates fall 6.4 pp on average — and the ranking flips.

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

The claim, up front

The paper is "RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests" (arXiv:2608.27831, published 2026-08-28, revised 2026-08-31).

Here is the abstract in plain terms. Coding agents are evaluated almost exclusively on the SWE-bench family, whose tasks are built from curated GitHub issues — long, structured, information-rich. Real user requests are far shorter and far less structured. To measure that gap, the authors define a six-category information taxonomy and four dimensions of linguistic style, and apply both to real user prompts from SWE-chat and to problem statements from SWE-bench Verified and Pro. Requests carrying only a problem statement, alone or with limited additional context, turn out to be 88% of real prompts but just 7% of benchmark problems. And 87% of real prompts are casually written, whereas 94% of benchmark problems are formal. From those observations they build RealSWE: 381 multi-variant task families derived from SWE-bench Verified and Pro, where variants inside a family share the same underlying task and the same gold patch and differ only in information composition and linguistic style. Evaluating seven contemporary LLMs, resolution rates drop by 6.4 percentage points on average and model rankings can change. Controlled analysis shows that including Desired Behavior and Motivation significantly affects performance, while Environment Information and Reproduction Steps merely add tokens with no measurable benefit; linguistic style has only small, model-dependent effects.

An analogy: describing the same fault to a mechanic, twice

Version one reads like a service log. "Above 60 km/h there is an abnormal noise while driving. It does not reproduce from cold. Last oil change was three months ago. The desired state is no noise at any speed."

Version two is a phone call. "Car's making a weird noise, fix it."

The fault is the same. The correct repair is the same. Only the telling differs — and yet the mechanic who gets the second version has to reconstruct, alone, where the noise comes from, when it started, and what "fixed" would even mean.

That is exactly the experiment. Build both tellings of the same underlying task, hand them to the same agent, and measure how much the fix rate moves. The starting observation of the paper (§1) is that conventional benchmarks only ever hand over version one.

How the paper measures a request

A request is described along two orthogonal axes (§3.1).

Axis one is the information taxonomy, grounded in GitHub's default issue templates and split by task type.

• Bug fix: [P] problem statement (observed failure) / [D] desired behavior / [R] reproduction steps / [E] environment information / [A] additional information
• Feature request: [P] problem statement (requested feature) / [M] motivation / [A] additional information

[A] absorbs everything the other categories do not cover — source URLs, issue-author metadata, dates. A request is then just the set of categories it contains: [P], [PA], [PDR], and so on.

Axis two is linguistic style, along four dimensions: formality, sentence type, certainty, and perspective.

The authors apply this ruler to 718 user-authored prompts from SWE-chat and to the 1,229 problem statements in SWE-bench Verified and Pro. SWE-chat is a public dataset of more than 6,000 real developer–agent sessions; because SWE-bench evaluates from a single problem statement with no further interaction, only the first user request of each session is kept (§3.1). GPT-5.4 does the annotation (§3.2).

What the measurement found

On information composition the two sources diverge sharply (§3.2):

• Real users: [P] and [PA] alone account for 88% of prompts (85.5% of bug fixes, 91.1% of feature requests)
• Benchmarks: [P] / [PA] account for 7% (8.0% of bug fixes, 3.9% of feature requests)

On language, 86.8% of real requests are casual and 51.3% are imperative. Against that, 84.8% of SWE-bench Verified prompts and 100% of Pro prompts are formal, and roughly 89% of both are declarative. Certainty and perspective, by contrast, show no consistent separation. So the gap is really about two things — how much information a request carries and how formal it sounds — not about all four style dimensions at once.

Why represent information as a set? Take the five bug-fix categories and assume [P] is always present. The remaining four are each present or absent, giving

In words, with the problem statement always present, each of the other four bug-fix categories can be either included or left out, so there are 16 possible combinations of information.

possible compositions. That arithmetic is mine, not the paper's; the paper simply notes that "the number of possible field combinations grows combinatorially" (§4.3). Multiply by the number of models and repeated runs and the evaluation budget stops being tractable — which is why the main analysis uses a single remove-one-field-at-a-time path rather than the full grid.

Building RealSWE

Knowing the gap, the next step is to express the same task differently. The pipeline has three stages (§3.3; full prompts in Appendix H).

1. Task-type classification. GPT-5.4 labels each problem as a bug fix or a feature request. Of the 1,231 problem statements in SWE-bench Verified and Pro, only 2 fit neither category and are dropped before the rest of the pipeline.
2. Information decomposition. Scaffolding inherited from issue templates — Markdown headings, HTML tags — is stripped while preserving user-written text. The problem statement is then segmented into sentences and each sentence is assigned to a taxonomy category. The text is redistributed, not rewritten, so decomposition does not change what the request carries.
3. Rephrasing in real-user style. GPT-5.4 rewrites the restructured problem conditioned on the majority category of each linguistic dimension measured in §3.2, while preserving technical literals: code blocks, error messages, tracebacks, file paths.

Assembling arbitrary compositions requires every taxonomy field to exist in the source problem statement. Of the 1,229 problems, 403 qualified.