#harness

11 articles

01 ·Paper Deep-Dives·★ MEMBER·PAPER·7 min read Paper Deep Dive: From Evidence to Action: How Tool-Using Agents Fail — Reading the Failure Diagnostics and Interventions SafeActBench splits tool-using agent failures into a chain of investigate, act, and hand off. This deep dive reads the paper through its appendices: how each failure diagnostic is defined, what agents do when decisive evidence is removed, and whether structuring evidence actually helps. 02 ·Agents·FREE·PAPER·6 min read SafeActBench explained (arXiv:2610.07753) — tool agents fail before execution A correct outcome does not mean the action was supported by evidence gathered beforehand. Across 656 SafeActBench cases and ten model-harness configurations, static action judgment scores 83.7-97.7 while interactive execution falls to 47.3-72.6 — and most failures start before execution. 03 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: WHALE — Training the Weights and the Harness in Turns An agent's performance is set by two things at once: the model weights, and the harness code wrapped around them. Polish one and the other becomes the ceiling. Here is a paper that dissolves that moving bottleneck by simply alternating two phases — explained from the metaphor through the math, the measured numbers, and the limits. 04 ·Agents·★ MEMBER·PAPER·12 min read HarnessDev explained (arXiv:2609.01437) — agent harness benchmark Identical weights, different harness: GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI. HarnessDev asks whether a model can build its own harness from a useless seed — and what survives held-out tasks. 05 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: Training Agents to Evolve with Their Harness Why compact models break when you rewrite prompts and tool definitions every week, and what the team behind Taobao Live's AI avatar streamers does about it. A ground-up walkthrough of Harness-Aware Training, from intuition to equations to measured numbers and stated limits. 06 ·Agents·★ MEMBER·PAPER·13 min read Paper Explained: StarHarness — Evolving the Scaffold Instead of the Weights Freeze the model weights and search over the harness instead — prompts, tool schemas, skills, MCP providers, subagents, agent loop. Across three enterprise benchmarks this bought 20–35 points, held on tasks never used for evolution, and transferred to other models unchanged. 07 ·Agents·★ MEMBER·PAPER·15 min read Paper Walkthrough: PILOT in the Loop — Fixing the Run While It Is Still Running Agent self-improvement that only starts after execution ends arrives too late. A ground-up walkthrough of PILOT, a supervisor–worker harness that redirects the live run while distilling skills from it, from intuition to equations to measured numbers and stated limits. 08 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: AutoSaddler — Growing a Harness That Doesn't Break, from Agent Failure Logs A ground-up walkthrough of AutoSaddler, which automatically optimizes the harness around an LLM agent — prompts, tools, and middleware — by repeatedly diagnosing failure traces and generating structured patches. It beat the base harnesses on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 by 9.0, 9.6, and 10.0 points. 09 ·Agents·★ MEMBER·PAPER·14 min read Paper Explained: JIT-Agent — A Model That Writes the Agent Harness On Demand An agent's capability is not the model's alone. This is a ground-up walkthrough of JIT-Agent, a model trained to synthesize the four-module harness — memory, planning, action, tools — freshly for each task. 10 ·Agents·FREE·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 11 ·Agents·FREE·PAPER·8 min read Paper Explained: LongHorizon-Harness — Long-Horizon Agent Tasks Are a State-Management Problem, Not an Execution Problem Without touching the model at all, keeping task state outside execution and running a Manage-Execute-Audit loop dramatically improves long-horizon task performance — a walkthrough of Alibaba DreamX team's harness-design paper, from analogy to pseudocode.