problems

Definition

A set of tasks with defined correct answers — maths word problems, competition programming items, unit-tested coding tasks. Benchmarks are built from them precisely because they can be scored automatically, and accuracy over a problem set is the usual proxy for capability. Distinct from the informal sense of "issues with an approach"; here a problem is a gradeable unit.

Explainers using this term

• Reward Hacking — Whatever You Measure Is Where It BreaksConcrete Problems in AI Safety