#benchmark

31 articles

01 ·Paper Deep-Dives·★ MEMBER·PAPER·7 min read Paper Deep Dive: From Evidence to Action: How Tool-Using Agents Fail — Reading the Failure Diagnostics and Interventions SafeActBench splits tool-using agent failures into a chain of investigate, act, and hand off. This deep dive reads the paper through its appendices: how each failure diagnostic is defined, what agents do when decisive evidence is removed, and whether structuring evidence actually helps. 02 ·Agents·FREE·PAPER·6 min read SafeActBench explained (arXiv:2610.07753) — tool agents fail before execution A correct outcome does not mean the action was supported by evidence gathered beforehand. Across 656 SafeActBench cases and ten model-harness configurations, static action judgment scores 83.7-97.7 while interactive execution falls to 47.3-72.6 — and most failures start before execution. 03 ·Training & Alignment·★ MEMBER·PAPER·7 min read Paper Explained: When Wording Changes an AI Reviewer's Score — RobustReview and SciCore A from-scratch walkthrough of a paper that defines 'rhetorical robustness' for AI reviewers whose scores shift when the same science is reworded. We cover 'false robustness', where stability alone fools you, and SciCore, which averages a full-manuscript judgment with a judgment of an extracted science core. 04 ·Large Language Models·★ MEMBER·PAPER·8 min read Paper Explained: Training Object Permanence in World Models — Teaching Video Models That Hidden Things Still Exist Video generators make hidden objects vanish and let balls pass through walls. We walk through WROP, a 150-task Blender benchmark for object permanence and solidity, the 16B PWM-WROP model fine-tuned on it, and how to read the human-preference evaluation behind the results. 05 ·Time Series·★ MEMBER·PAPER·13 min read Paper Review: MetroLLM-Bench — What Happens When an LLM Runs the Ticket Machine. Why a 2.6 GB 4B Model Tied GPT-5.6, and Why Fine-Tuning Hurt at 27B A 955-case benchmark that asks how well a language model can serve as the decision layer of a transit ticket kiosk. We read, from the paper alone, how a 4B QLoRA student beat GPT-5.6 on the deterministic tier and why PEFT gains shrink with model size until they turn negative at 27B. 06 ·Time Series·★ MEMBER·PAPER·11 min read Paper Explained: WearableQA — Testing Whether LLMs Can Reason Over 500 Days of a Real User's Wearable Data 4,084 ten-option questions built from 200 real users' daily wearable records (up to 500 days each) plus blood panels. Separating 'answer from knowledge' from 'read it off the numbers' splits 14 models between 19.6% and 72.9% accuracy, and simply giving the model Python adds 20 points. A ground-up walkthrough of how the questions are built and what the results actually say. 07 ·★ MEMBER·PAPER·11 min read Paper Explained: VDiff-Bench — Even Frontier MLLMs Miss the Low-Level Changes in a Spot-the-Difference Puzzle VDiff-Bench asks a multimodal LLM what changed between two near-identical images, as a four-way multiple choice. We walk through how its 1,756 questions across 10 change categories were built, what 11 models scored, why several of them answer 'no difference' when noise or texture changed, and what this means if you use an MLLM to grade image edits. 08 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: S³Gym — Does Self-Testing and Self-Judging Add Up to Self-Improvement? Can an LLM agent really learn from its own experience? A benchmark that separates permissive exploration from strict held-out evaluation across seven games, comparing raw history, summary memory, and parameter training under one protocol. 09 ·Large Language Models·★ MEMBER·PAPER·12 min read Paper walkthrough: RealSWE — re-measuring coding agents when the request is just "fix it" A paper that measures, on real data, how far SWE-bench problem statements sit from actual user requests, then builds 381 task families where only the information content and writing style change. Resolution rates fall 6.4 pp on average — and the ranking flips. 10 ·Agents·★ MEMBER·PAPER·12 min read HarnessDev explained (arXiv:2609.01437) — agent harness benchmark Identical weights, different harness: GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 and 49.6% inside Codex CLI. HarnessDev asks whether a model can build its own harness from a useless seed — and what survives held-out tasks. 11 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: EarlyEval — Making Agent Evaluation Cheaper by Stopping Early One evaluation pass over an agentic benchmark costs hundreds of dollars. EarlyEval exploits the fact that an agent's ending is legible from its middle, halting runs to cut 13–26% of steps. Mechanism, numbers, and limits, straight from the paper. 12 ·Agents·★ MEMBER·PAPER·14 min read Paper Walkthrough: Aspire — Can Models Self-Evolve from Vague Goals? Hand an agent nothing but "get better at mathematical reasoning" and let it decide what to learn and how to check itself. Aspire measures the result on 520 hidden, expert-written items — and finds that agents close the training loop far more reliably than the capability loop. 13 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough — UrbanGround: Where MLLM Agents Break Down on a Real Street Drop an MLLM agent into a real-scale replica of Hong Kong built from territory-wide 3D geospatial data. Visual recognition clears 90%, orientation sits near 40%, long-range navigation is close to 0%. A walkthrough of the benchmark that measures the gap between seeing and moving. 14 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding Agent A ground-up walkthrough of LoopArena, a benchmark that scores the outer loop steering a coding agent while holding the agent itself fixed — including how to read the headline result of 24.69% strict success on full tasks. 15 ·Evaluation & Judging·FREE·PAPER·10 min read LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels. 16 ·Evaluation & Judging·★ MEMBER·PAPER·11 min read Benchmark Contamination — How to Doubt a High Score How to tell whether a benchmark score reflects ability or memorization. Covers the three kinds of contamination, the three detection tools — n-gram overlap, embedding neighbors, membership inference — and where each stops working, plus canary strings and time-based splits, ending in a checklist for reading papers skeptically. 17 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 18 ·Training & Alignment·★ MEMBER·PAPER·10 min read PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics? If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once. 19 ·FREE·PAPER·9 min read Paper Explained: VGI-Bench — Can Video Models Get the Middle of the Story Right? A ground-up walkthrough of VGI-Bench, a benchmark that treats video generation models as machines that reason in pictures. Even the strongest model, Seedance 2.0, scores only 51.0 — and self-correction during denoising almost never happens. 20 ·FREE·PAPER·15 min read Paper Explained: VBVR-Pro — Making "Thinking in Pictures" Trainable, Scorable, and Comparable Native visual reasoning treats image and video generation as the medium of thought. VBVR-Pro supplies 300 procedurally generated tasks and deterministic scorers that replace the VLM judge, then pits image, video, and interleaved generation against each other on identical ground. Written for readers starting from zero. 21 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: FrontierChallenge — Grading Scientific Work on Whether It Was Actually Delivered A walkthrough of FrontierChallenge, a benchmark that scores whether an agent finishes a scientific workflow. Average scores hit 87.9 while full completion stayed at 20.6% — and 0% in electrochemistry, where the average was 94.9. Worse, 75.5% of failing trajectories still ended by claiming they were done. 22 ·Large Language Models·FREE·PAPER·11 min read LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge. 23 ·Agents·★ MEMBER·PAPER·10 min read Paper walkthrough: ASI-Bench — peeling away human guidance to measure what AI can do alone ASI-Bench keeps the research goal, data and grading fixed while stripping away human methodological guidance one layer at a time. Average scores fall 50.91 → 29.10 → 26.62, and the place where things break is not method selection. 24 ·Agents·FREE·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 25 ·FREE·PAPER·11 min read Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?' SemComp-Bench moves video-generation evaluation from 'does it look good' to 'did it reach the outcome I asked for'. A walkthrough of the task definition, the four-stage data pipeline, the AND-vs-average scoring, and the measured numbers — all from the paper itself. 26 ·Agents·★ MEMBER·PAPER·13 min read Paper Explainer: SemaPLC — The Agent That Isn't Allowed to Say "Done" Most work on LLM-generated factory control code stops at showing that the code can run. SemaPLC forbids the agent from declaring completion until logged external checks confirm it, lifting live-runtime behavior from a baseline ceiling of 31.4 to 52.2. 27 ·Large Language Models·★ MEMBER·PAPER·8 min read Paper Walkthrough: Can Anything Catch a Fake Crisis Video? — What RA-Bench Found Sixteen thousand AI videos, each continuing from the real first frame of a genuine disaster or war clip, put against seven classical detectors, ten zero-shot multimodal models and two purpose-built fine-tunes. None of them generalized. One model turned out to be reading timestamps rather than pixels, and a lap through a social feed drops fake recall to 1.4%. 28 ·Agents·FREE·PAPER·10 min read Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans Do A close read of MerchantBench, which measures the long-term coherence of LLM agents across a 365-day simulation of running an online store. Why the best configuration reaches only 27.3% of the average human's final net worth, read through three failure patterns: decaying activity, premature withdrawal, and policies frozen against the evidence. 29 ·FREE·PAPER·8 min read Paper Walkthrough: GST-Bench — Can VLMs Build a Global Map of a Scene from Video? Show a VLM a walkthrough video of a house, then ask 'from where you're standing now, which way is the sofa?' — even the strongest model scores barely half of what humans do. A walkthrough of GST-Bench from ByteDance Seed: the shortcut-proof benchmark design, results across 22 models, and the training data that closed 27 points of the gap. 30 ·Inference & Serving·FREE·PAPER·9 min read Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most. 31 ·Agents·FREE·PAPER·7 min read Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.