live

Definition

Continuously updated rather than fixed. Live benchmarks add fresh problems on a rolling schedule specifically to limit contamination — the situation where test items have leaked into training data and inflate scores for models released later. The trade-off is comparability: because the question set changes, numbers from different snapshots are not directly comparable, so results should state which version was used.

Explainers using this term

• Paper Walkthrough: PILOT in the Loop — Fixing the Run While It Is Still RunningPILOT in the Loop: Live Self-Improvement for Long-Horizon Agents
• Paper Walkthrough: TLive-Omni — An Omni-Modal Model That Watches, Listens, and Tells You the TimestampTLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming