Students Professional
JA EN

#benchmark

24 articles

01 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification Rules Machine translation benchmarks are saturating, and neither automatic metrics nor human evaluation can be fully trusted. The response: collect human-written examples that break frontier models, and attach handcrafted verification rules to each one. A ground-up reading of Last Translation Benchmark. 02 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: HarnessDev — Can an LLM Build and Maintain the System It Runs Inside? A ground-up walkthrough of HarnessDev, a benchmark that asks whether an LLM can build an agent harness from a deliberately useless seed and then improve it from execution feedback. Models can build one; the gains rarely survive contact with held-out tasks or a different runtime model. 03 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: EarlyEval — Making Agent Evaluation Cheaper by Stopping Early One evaluation pass over an agentic benchmark costs hundreds of dollars. EarlyEval exploits the fact that an agent's ending is legible from its middle, halting runs to cut 13–26% of steps. Mechanism, numbers, and limits, straight from the paper. 04 ·Agents·★ MEMBER·PAPER·14 min read Paper Walkthrough: Aspire — Can Models Self-Evolve from Vague Goals? Hand an agent nothing but "get better at mathematical reasoning" and let it decide what to learn and how to check itself. Aspire measures the result on 520 hidden, expert-written items — and finds that agents close the training loop far more reliably than the capability loop. 05 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough — UrbanGround: Where MLLM Agents Break Down on a Real Street Drop an MLLM agent into a real-scale replica of Hong Kong built from territory-wide 3D geospatial data. Visual recognition clears 90%, orientation sits near 40%, long-range navigation is close to 0%. A walkthrough of the benchmark that measures the gap between seeing and moving. 06 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding Agent A ground-up walkthrough of LoopArena, a benchmark that scores the outer loop steering a coding agent while holding the agent itself fixed — including how to read the headline result of 24.69% strict success on full tasks. 07 ·Evaluation & Judging·FREE·PAPER·10 min read LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels. 08 ·Evaluation & Judging·★ MEMBER·PAPER·10 min read Benchmark Contamination — How to Doubt a High Score How to tell whether a benchmark score reflects ability or memorization. Covers the three kinds of contamination, the three detection tools — n-gram overlap, embedding neighbors, membership inference — and where each stops working, plus canary strings and time-based splits, ending in a checklist for reading papers skeptically. 09 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 10 ·Training & Alignment·★ MEMBER·PAPER·10 min read PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics? If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once. 11 ·★ MEMBER·PAPER·9 min read Paper Explained: VGI-Bench — Can Video Models Get the Middle of the Story Right? A ground-up walkthrough of VGI-Bench, a benchmark that treats video generation models as machines that reason in pictures. Even the strongest model, Seedance 2.0, scores only 51.0 — and self-correction during denoising almost never happens. 12 ·★ MEMBER·PAPER·14 min read Paper Explained: VBVR-Pro — Making "Thinking in Pictures" Trainable, Scorable, and Comparable Native visual reasoning treats image and video generation as the medium of thought. VBVR-Pro supplies 300 procedurally generated tasks and deterministic scorers that replace the VLM judge, then pits image, video, and interleaved generation against each other on identical ground. Written for readers starting from zero. 13 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: FrontierChallenge — Grading Scientific Work on Whether It Was Actually Delivered A walkthrough of FrontierChallenge, a benchmark that scores whether an agent finishes a scientific workflow. Average scores hit 87.9 while full completion stayed at 20.6% — and 0% in electrochemistry, where the average was 94.9. Worse, 75.5% of failing trajectories still ended by claiming they were done. 14 ·Agents·★ MEMBER·PAPER·9 min read Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code? A walkthrough of SWE-bench Science, a repository-level benchmark of 119 tasks from 98 repos across 20 scientific domains. The best agent scores under 50% pass@1, and the paper names four recurring failure mechanisms plus an ablation showing scientific knowledge is not uniformly helpful. 15 ·Large Language Models·★ MEMBER·PAPER·11 min read LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge. 16 ·Agents·★ MEMBER·PAPER·10 min read Paper walkthrough: ASI-Bench — peeling away human guidance to measure what AI can do alone ASI-Bench keeps the research goal, data and grading fixed while stripping away human methodological guidance one layer at a time. Average scores fall 50.91 → 29.10 → 26.62, and the place where things break is not method selection. 17 ·Agents·★ MEMBER·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 18 ·★ MEMBER·PAPER·11 min read Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?' SemComp-Bench moves video-generation evaluation from 'does it look good' to 'did it reach the outcome I asked for'. A walkthrough of the task definition, the four-stage data pipeline, the AND-vs-average scoring, and the measured numbers — all from the paper itself. 19 ·Agents·★ MEMBER·PAPER·13 min read Paper Explainer: SemaPLC — The Agent That Isn't Allowed to Say "Done" Most work on LLM-generated factory control code stops at showing that the code can run. SemaPLC forbids the agent from declaring completion until logged external checks confirm it, lifting live-runtime behavior from a baseline ceiling of 31.4 to 52.2. 20 ·Large Language Models·★ MEMBER·PAPER·8 min read Paper Walkthrough: Can Anything Catch a Fake Crisis Video? — What RA-Bench Found Sixteen thousand AI videos, each continuing from the real first frame of a genuine disaster or war clip, put against seven classical detectors, ten zero-shot multimodal models and two purpose-built fine-tunes. None of them generalized. One model turned out to be reading timestamps rather than pixels, and a lap through a social feed drops fake recall to 1.4%. 21 ·Agents·★ MEMBER·PAPER·10 min read Paper Walkthrough: MerchantBench — Can an LLM Agent Run an Online Store for a Year? Why It Earns Only 27.3% of What Humans Do A close read of MerchantBench, which measures the long-term coherence of LLM agents across a 365-day simulation of running an online store. Why the best configuration reaches only 27.3% of the average human's final net worth, read through three failure patterns: decaying activity, premature withdrawal, and policies frozen against the evidence. 22 ·★ MEMBER·PAPER·8 min read Paper Walkthrough: GST-Bench — Can VLMs Build a Global Map of a Scene from Video? Show a VLM a walkthrough video of a house, then ask 'from where you're standing now, which way is the sofa?' — even the strongest model scores barely half of what humans do. A walkthrough of GST-Bench from ByteDance Seed: the shortcut-proof benchmark design, results across 22 models, and the training data that closed 27 points of the gap. 23 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most. 24 ·Agents·★ MEMBER·PAPER·7 min read Paper Walkthrough: SWE-Bench ProMax — Measuring What Coding Agents Can Really Do with Large-Scale, Multilingual Refactoring A close read of the benchmark built to answer SWE-bench's saturation and grading defects: 170 expert-curated refactoring tasks across seven languages, averaging 11.4 modified files each. Why the best frontier model resolves only 41.2%, and why the dominant failure mode is refactoring that stops halfway.