Students Professional
JA EN

#evaluation

48 articles

01 ·Audio & Speech·★ MEMBER·PAPER·9 min read Paper Explained: Last Translation Benchmark — Measuring Translation with Breaking Examples and Verification Rules Machine translation benchmarks are saturating, and neither automatic metrics nor human evaluation can be fully trusted. The response: collect human-written examples that break frontier models, and attach handcrafted verification rules to each one. A ground-up reading of Last Translation Benchmark. 02 ·Agents·★ MEMBER·PAPER·9 min read CogEvol: What the Reward Cannot Measure, RL Will Quietly Destroy A technical report on a model family that generates teaching material in a single pass. Its centerpiece is an incident the authors disclose in full: a screenshot-only reward taught the policy to ship games that looked convincing and could not be played. 03 ·Agents·★ MEMBER·PAPER·12 min read Paper Walkthrough: EarlyEval — Making Agent Evaluation Cheaper by Stopping Early One evaluation pass over an agentic benchmark costs hundreds of dollars. EarlyEval exploits the fact that an agent's ending is legible from its middle, halting runs to cut 13–26% of steps. Mechanism, numbers, and limits, straight from the paper. 04 ·Agents·★ MEMBER·PAPER·14 min read Paper Walkthrough: Aspire — Can Models Self-Evolve from Vague Goals? Hand an agent nothing but "get better at mathematical reasoning" and let it decide what to learn and how to check itself. Aspire measures the result on 520 hidden, expert-written items — and finds that agents close the training loop far more reliably than the capability loop. 05 ·Training & Alignment·★ MEMBER·PAPER·12 min read Paper Walkthrough: StudentSim — Training a Simulator That Is Actually *That* Student How to build the practice students an AI tutor needs, by training on real learner records. From the two-axis definition of behavioral fidelity and guidance responsiveness, through the pooled-then-specialized pipeline, to using the simulator as a reward model. 06 ·Agents·★ MEMBER·PAPER·13 min read Paper Explained: StarHarness — Evolving the Scaffold Instead of the Weights Freeze the model weights and search over the harness instead — prompts, tool schemas, skills, MCP providers, subagents, agent loop. Across three enterprise benchmarks this bought 20–35 points, held on tasks never used for evolution, and transferred to other models unchanged. 07 ·Evaluation & Judging·★ MEMBER·PAPER·12 min read Self-Improving AI — Self-Play, Co-Evolution, and Generated Curricula What has to be true for a model to get better without anyone adding data? This article pulls three conditions out of AlphaZero's self-play, shows exactly which one breaks first for language models, explains how co-evolution and generated curricula try to patch the gap, and ends with why self-improvement claims are unusually easy to evaluate wrong. 08 ·Evaluation & Judging·★ MEMBER·PAPER·10 min read Reward Hacking — Whatever You Measure Is Where It Breaks The moment you pick a metric, that metric starts to rot. This piece explains why Goodhart's law is statistically unavoidable, walks through real failures from boat races that spin in circles to RLHF verbosity, sycophancy and hardcoded unit tests, and covers how to detect the gap between optimization pressure and true performance. 09 ·Large Language Models·★ MEMBER·PAPER·13 min read Paper walkthrough: Puro-2B — pretraining a 2B model from scratch for $6.9K on consumer GPUs A team ran 1.4 trillion tokens of pretraining on gaming GPUs and reached Qwen2-1.5B-level quality for roughly $4.4K. Here is the cost structure, the FP8 accounting, the effective learning rate, and the curriculum averaging — from first principles. 10 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: LoopArena — Benchmarking the Model That Steers a Coding Agent A ground-up walkthrough of LoopArena, a benchmark that scores the outer loop steering a coding agent while holding the agent itself fixed — including how to read the headline result of 24.69% strict success on full tasks. 11 ·Evaluation & Judging·FREE·PAPER·10 min read LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels. 12 ·Evaluation & Judging·FREE·PAPER·10 min read LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels. 13 ·Evaluation & Judging·★ MEMBER·PAPER·10 min read Benchmark Contamination — How to Doubt a High Score How to tell whether a benchmark score reflects ability or memorization. Covers the three kinds of contamination, the three detection tools — n-gram overlap, embedding neighbors, membership inference — and where each stops working, plus canary strings and time-based splits, ending in a checklist for reading papers skeptically. 14 ·Evaluation & Judging·★ MEMBER·PAPER·10 min read Benchmark Contamination — How to Doubt a High Score How to tell whether a benchmark score reflects ability or memorization. Covers the three kinds of contamination, the three detection tools — n-gram overlap, embedding neighbors, membership inference — and where each stops working, plus canary strings and time-based splits, ending in a checklist for reading papers skeptically. 15 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 16 ·Evaluation & Judging·FREE·PAPER·11 min read A Map of Agent Benchmarks — What SWE-bench, GAIA, and OSWorld Actually Measure What the three most-cited agent benchmarks — SWE-bench, GAIA, and OSWorld — really measure, read off from how each one grades. Includes the four distinct kinds of contamination and a checklist for translating a published score into something useful for your own system. 17 ·Large Language Models·★ MEMBER·PAPER·9 min read Paper Explained: Agentic Artifact Creation — Where Generation Ends and Construction Begins A survey that reorganizes 259 works around a single unit: the delivered artifact. It defines agentic creation through state, edits, and verification, then works through six artifact families, three evaluation targets, four principles, and six open problems — from first principles. 18 ·Large Language Models·★ MEMBER·PAPER·9 min read Paper Explained: Agentic Artifact Creation — Where Generation Ends and Construction Begins A survey that reorganizes 259 works around a single unit: the delivered artifact. It defines agentic creation through state, edits, and verification, then works through six artifact families, three evaluation targets, four principles, and six open problems — from first principles. 19 ·Training & Alignment·★ MEMBER·PAPER·10 min read PAWBench Explained — Can Video Generators Get the Odds Right, Not Just the Physics? If a video generator is a world model, it owes you more than one plausible rollout — it owes you the right distribution over futures. PAWBench measures that probabilistic alignment across 50 scenarios and 11 systems, and finds that no model gets all the requirements at once. 20 ·Agents·★ MEMBER·PAPER·13 min read Paper Explained: JIT-Agent — A Model That Writes the Agent Harness On Demand An agent's capability is not the model's alone. This is a ground-up walkthrough of JIT-Agent, a model trained to synthesize the four-module harness — memory, planning, action, tools — freshly for each task. 21 ·★ MEMBER·PAPER·9 min read Paper Explained: VGI-Bench — Can Video Models Get the Middle of the Story Right? A ground-up walkthrough of VGI-Bench, a benchmark that treats video generation models as machines that reason in pictures. Even the strongest model, Seedance 2.0, scores only 51.0 — and self-correction during denoising almost never happens. 22 ·Agents·★ MEMBER·PAPER·8 min read Paper Walkthrough: FrontierChallenge — Grading Scientific Work on Whether It Was Actually Delivered A walkthrough of FrontierChallenge, a benchmark that scores whether an agent finishes a scientific workflow. Average scores hit 87.9 while full completion stayed at 20.6% — and 0% in electrochemistry, where the average was 94.9. Worse, 75.5% of failing trajectories still ended by claiming they were done. 23 ·Distillation & Compression·★ MEMBER·PAPER·9 min read Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric? Score a distilled student by how often it agrees with its teacher and the students that faithfully reproduce the teacher's mistakes come out on top. What agreement actually guarantees, what breaks outside the training distribution, and how to test for contamination that arrives paraphrased through synthetic data. 24 ·Distillation & Compression·★ MEMBER·PAPER·9 min read Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric? Score a distilled student by how often it agrees with its teacher and the students that faithfully reproduce the teacher's mistakes come out on top. What agreement actually guarantees, what breaks outside the training distribution, and how to test for contamination that arrives paraphrased through synthetic data. 25 ·Agents·★ MEMBER·PAPER·9 min read Paper Walkthrough: SWE-bench Science — Can Coding Agents Fix Scientific Code? A walkthrough of SWE-bench Science, a repository-level benchmark of 119 tasks from 98 repos across 20 scientific domains. The best agent scores under 50% pass@1, and the paper names four recurring failure mechanisms plus an ablation showing scientific knowledge is not uniformly helpful. 26 ·RAG & Retrieval·★ MEMBER·PAPER·13 min read Evaluating RAG in Practice — Turning “Seems Better” Into a Number RAG improvement work gets lost because quality is judged as one vague feeling. This article builds up the metrics that separate retrieval from generation, the definitions of faithfulness and relevance, how to build an eval set with synthetic QA and where that goes wrong, and how many questions you need before the number means anything. 27 ·RAG & Retrieval·★ MEMBER·PAPER·13 min read Evaluating RAG in Practice — Turning “Seems Better” Into a Number RAG improvement work gets lost because quality is judged as one vague feeling. This article builds up the metrics that separate retrieval from generation, the definitions of faithfulness and relevance, how to build an eval set with synthetic QA and where that goes wrong, and how many questions you need before the number means anything. 28 ·Agents·★ MEMBER·PAPER·10 min read Multi-Agent Design Patterns — Division, Debate, Verification Stack as many agents as you like — if they all fail the same way, you have one agent and a larger bill. The condition under which voting actually helps, written down, then the three patterns that follow from it: division of labour, debate, and adversarial verification — plus when one agent is enough. 29 ·How Transformers Work·★ MEMBER·PAPER·11 min read How Long-Context LLMs Work — From RoPE Interpolation to Ring Attention A "128K context window" is two different walls, knocked down by two unrelated families of tricks. This walks through positional interpolation, NTK-aware scaling and YaRN for the position wall; sliding windows and ring attention for the compute wall; and how to read a needle-in-a-haystack chart without being fooled by it. 30 ·Large Language Models·★ MEMBER·PAPER·11 min read LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge. 31 ·Large Language Models·★ MEMBER·PAPER·11 min read LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge. 32 ·Machine Learning Basics·★ MEMBER·10 min read Imbalanced Data in Practice — What to Optimize When 99% Is Normal When only 1% of your data is positive, accuracy lies. We build up from the confusion matrix to PR curves, class weighting, resampling, probability calibration, and deriving the decision threshold from cost — no prior knowledge assumed. 33 ·Machine Learning Basics·★ MEMBER·10 min read Imbalanced Data in Practice — What to Optimize When 99% Is Normal When only 1% of your data is positive, accuracy lies. We build up from the confusion matrix to PR curves, class weighting, resampling, probability calibration, and deriving the decision threshold from cost — no prior knowledge assumed. 34 ·Large Language Models·FREE·PAPER·9 min read Why Language Models Hallucinate — The Mechanics and What Actually Helps Confident fabrication isn't a bug — it falls straight out of next-token prediction. This piece takes the mechanism apart layer by layer: no truth term in the loss, lossy compression that fails at the edges, sampling that draws from the tail, and scoring rules that pay models to guess. Then it ranks the countermeasures that actually work: grounding, mechanical citation checking, constrained output, and uncertainty estimation. 35 ·Machine Learning Basics·★ MEMBER·10 min read Data Leakage and Experiment Hygiene — When the Score Is Too Good, Suspect It When a model scores better than you expected, that is not good news yet. This article splits data leakage into five concrete shapes — outcome-derived columns, preprocessing before the split, time running backwards, duplicates and groups, and a worn-out test set — then covers the detection moves that actually find them and the experiment records that let you dissect the day the number jumped. 36 ·Machine Learning Basics·★ MEMBER·10 min read Data Leakage and Experiment Hygiene — When the Score Is Too Good, Suspect It When a model scores better than you expected, that is not good news yet. This article splits data leakage into five concrete shapes — outcome-derived columns, preprocessing before the split, time running backwards, duplicates and groups, and a worn-out test set — then covers the detection moves that actually find them and the experiment records that let you dissect the day the number jumped. 37 ·Agents·★ MEMBER·PAPER·10 min read Paper walkthrough: ASI-Bench — peeling away human guidance to measure what AI can do alone ASI-Bench keeps the research goal, data and grading fixed while stripping away human methodological guidance one layer at a time. Average scores fall 50.91 → 29.10 → 26.62, and the place where things break is not method selection. 38 ·Agents·★ MEMBER·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 39 ·Agents·★ MEMBER·PAPER·11 min read Evaluating Agents — How Benchmarks and Harnesses Are Built An agent score is never a property of the model alone — it is a property of model plus harness plus environment plus grading rule. This piece opens up a single SWE-bench instance, shows why a tiny per-step gap becomes an order-of-magnitude gap over a long horizon, walks the four routes by which the answer leaks into the working environment, and sets out the conditions under which partial credit is safe. 40 ·★ MEMBER·PAPER·11 min read Paper Walkthrough: SemComp-Bench — Asking Video Generators 'So Did It Actually Get Done?' SemComp-Bench moves video-generation evaluation from 'does it look good' to 'did it reach the outcome I asked for'. A walkthrough of the task definition, the four-stage data pipeline, the AND-vs-average scoring, and the measured numbers — all from the paper itself. 41 ·Large Language Models·★ MEMBER·PAPER·11 min read Paper Walkthrough: SA-MRPO — Stop Studying the Subject You've Already Aced When you train with several reward objectives at once, gradient budget keeps flowing to objectives that are already solved. SA-MRPO (arXiv:2608.16072) measures how saturated each objective is and discounts its weight accordingly, redirecting optimization toward whatever headroom remains. A walkthrough grounded strictly in the paper. 42 ·Large Language Models·★ MEMBER·PAPER·8 min read Paper Walkthrough: Can Anything Catch a Fake Crisis Video? — What RA-Bench Found Sixteen thousand AI videos, each continuing from the real first frame of a genuine disaster or war clip, put against seven classical detectors, ten zero-shot multimodal models and two purpose-built fine-tunes. None of them generalized. One model turned out to be reading timestamps rather than pixels, and a lap through a social feed drops fake recall to 1.4%. 43 ·Agents·★ MEMBER·PAPER·11 min read Paper Walkthrough: ComBodied Agents — Moving an Agent's Target from Software and Matter to the Person Digital agents rewrite software state; embodied agents rewrite physical state. So who models the person? Combodied Agents bind perception, longitudinal memory, a Personal World Model, and an intervention policy into one closed loop, and move the success criterion from task completion to whether the user's agency survived. A position paper with no experiments — here is its skeleton, and its limits. 44 ·Inference & Serving·★ MEMBER·PAPER·9 min read Paper Walkthrough: The Personalization Mirage — LLMs Invent a Version of You, and Their Self-Reports Point the Wrong Way How much of what a memory-equipped LLM 'knows' about you was never actually said? MirageBench judges 143,616 claims across 150 personas and 6 tasks, finding that all 12 tested models over-infer 35–49% of their claims — and that the models reporting the least over-inference tend to fabricate the most. 45 ·Time Series·★ MEMBER·9 min read Time-Series Forecasting from Scratch — From Classical Methods to Foundation Models What makes forecasting different from ordinary regression is that the rows are not independent and the test set is always in the future. Autocorrelation and stationarity first, then the intuition behind ARIMA, seasonal and trend decomposition, and where deep and foundation models actually sit. The last third is evaluation — no random splits, how to backtest, and why MAPE lies. 46 ·Large Language Models·FREE·PAPER·7 min read The Science of Prompt Engineering — What Is Proven and What Is Folklore Does telling a model to think step by step actually work? Working only from the Chain-of-Thought paper (Wei et al., 2022): what the experiments genuinely established — an ability that emerges only above a scale threshold — and where the line falls between that and the incantations everyone repeats. 47 ·Machine Learning Basics·★ MEMBER·8 min read Overfitting and Evaluation Design — Be Suspicious of 99% Accuracy Why a model that fits its training data perfectly falls apart in production: the division of labour between train, validation and test, cross-validation, the data leakage that ruins more projects than any modelling mistake, and how accuracy lies on imbalanced data. 48 ·Machine Learning Basics·★ MEMBER·8 min read Overfitting and Evaluation Design — Be Suspicious of 99% Accuracy Why a model that fits its training data perfectly falls apart in production: the division of labour between train, validation and test, cross-validation, the data leakage that ruins more projects than any modelling mistake, and how accuracy lies on imbalanced data.