livecodebench

Definition

A benchmark for code generation that continuously ingests fresh problems from competitive programming sites. Each problem carries a release date, so evaluations can be restricted to problems published after a model's training cutoff — a direct answer to contamination, which has quietly inflated scores on older static benchmarks. It also spans several task settings beyond writing a solution, including self-repair, code execution, and test output prediction.