overoptimization

Definition

What happens when a proxy is pushed too far: the measured score keeps climbing while the thing it was meant to stand for gets worse. It is the central failure mode of reward-model-based training, where the policy learns the quirks and blind spots of the grader instead of the intended behavior. Distinct from overfitting, which is about fitting the training data too closely; here the target itself is the problem.