reward

Definition

The scalar signal in reinforcement learning that tells the agent how good an action or outcome was; the policy is updated to increase expected reward. For language models the reward often comes from a learned reward model trained on human preference comparisons, standing in for expensive human judgement. Because policies optimize exactly what is measured, poorly specified rewards invite reward hacking — high scores achieved by behaviour nobody wanted.

Explainers using this term

• Paper Explained: RewardVerse — A Video Reward Model That Writes the Rubric Before It Scores (RGPO)RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling
• Paper Explained: Beyond Solver Verdicts — When the Solver Says Yes but the Translation Is WrongBeyond Solver Verdicts: Generative Reward Models for Autoformalization
• DPO and What Came After — The Lineage That Simplified RLHFDirect Preference Optimization: Your Language Model is Secretly a Reward Model