Reward Hacking — Whatever You Measure Is Where It Breaks

The moment you pick a metric, that metric starts to rot. This piece explains why Goodhart's law is statistically unavoidable, walks through real failures from boat races that spin in circles to RLHF verbosity, sycophancy and hardcoded unit tests, and covers how to detect the gap between optimization pressure and true performance.

Concrete Problems in AI Safety

The day someone put a bounty on cobras

Colonial India, the story goes, tried to cut its cobra population by paying a bounty for dead snakes. People stopped hunting cobras and started breeding them. The history is shaky, but the story survives because everyone has watched the same thing happen at work.

Grade a support team on average response time and someone starts closing hard tickets early. Grade a hospital on waiting time and a second queue appears in front of reception. What you measure is not the goal — it is a shadow of the goal, and there are many ways to enlarge a shadow that have nothing to do with achieving anything.

Reward hacking is this same phenomenon happening inside a model, faster and far more thoroughly than any human would manage. You have no channel for telling a model what you actually want; all you can hand it is a way of assigning points. And optimization is, by definition, the procedure that searches exhaustively for ways to maximize points. If a shortcut exists, it gets found. It would be strange if it didn't.

Goodhart's law — why it can't be avoided

Charles Goodhart's observation is best known in Marilyn Strathern's phrasing: when a measure becomes a target, it ceases to be a good measure.

This isn't a slogan; it's almost writable as an equation. Call the thing you actually want to maximize $U$ (true utility), and the thing you can actually measure $\tilde{U}$ (the proxy).

In words: the metric you can read off is what you really want, plus an error term $X$. Think of $X$ as bundling together measurement noise, the aspects the metric fails to capture, and the room left for shortcuts.

In everyday operation $X$ is small, so anything that scores high on $\tilde{U}$ is also high on $U$. That's why metrics are useful at all. The trouble is that optimization pushes you outside "everyday". When you hunt for points where $\tilde{U}$ is extreme, you stop selecting points that score high because $U$ is high and start selecting points that score high because $X$ is large. $U$ is bounded and saturates; $X$ can keep climbing for as long as the shortcut holds out.

So reward hacking isn't a symptom of a dishonest model. It's a statistical property of selection: the harder you optimize, the more the winning strategy shifts from the real thing to the error term. Correlations do not survive optimization.

Correlation only holds inside the normal range

This divergence has exactly the shape of the overfitting picture. Keep driving training error (the metric you're watching) down and, past a certain point, test error (the performance you actually want) starts climbing. Drag the degree slider below to the right and find where the two curves separate. That point is the edge of the range in which your metric can be trusted.

The crucial part is that the turn is invisible from the metric's side. The reward number keeps climbing beautifully right to the end. The mechanics of this are covered in overfitting and evaluation, but reward design has one extra difficulty: nobody computes the "test error" for you automatically.

Failures come in types

Before arguing about fixes, it helps to split the phenomenon into four kinds.

• Regressional: every proxy carries noise, so selecting the top of the distribution mixes in candidates that got lucky rather than candidates that are good. This is why best-of-N sampling keeps raising the reward model's score while human ratings stall.
• Extremal: a reward model is only calibrated near the responses humans actually compared. Wander outside that region and it will happily hand out high scores to bizarre output.
• Causal: optimizing a metric that mistook correlation for causation. "Good answers tend to be long" slides into "make it long and it will be good," and padding gets mass-produced.
• Adversarial: attacking the grading machinery itself — special-casing test inputs, responding to instructions smuggled into an evaluation prompt.

These need different prescriptions. The first two are "control the strength of optimization" problems, the third is a "redesign the metric" problem, and the fourth is "defend the grader against attack." Discuss them together and a question you can fix with a coefficient ends up at the same table as one where the metric has to be thrown out.