Paper Explained: Refusal in Language Models Is Mediated by a Single Direction — saying "no" lives on one line inside the model

When chat models turn down harmful requests, that behavior is carried by a single direction in the residual stream. This walkthrough reads the paper as an interpretability result: how a difference of mean activations finds the direction, why removing it stops refusal, and why adding it makes models refuse harmless requests across 13 models.

Refusal in Language Models Is Mediated by a Single Direction

What the paper did

The paper is "Refusal in Language Models Is Mediated by a Single Direction" by Andy Arditi, Oscar Obeso, and five co-authors (arXiv:2406.11717).

Here is the abstract in brief. Chat models are fine-tuned to follow instructions and to be safe, so they help with benign requests and refuse harmful ones. Refusal is everywhere, but how it works inside the model was poorly understood. Across 13 popular open-source chat models up to 72B parameters, the authors show that refusal is mediated by a one-dimensional subspace. For each model there is a single direction such that erasing it from the residual stream stops the model from refusing harmful instructions, while adding it makes the model refuse even harmless ones. Building on this, they propose a white-box jailbreak that disables refusal with little effect on other capabilities, and they analyze how adversarial suffixes (strings appended to a prompt to break safety) suppress the propagation of this direction. They conclude that current safety fine-tuning is brittle, and that understanding model internals can yield practical ways to control behavior.

This article stays on the interpretability side. It does not describe how to strip safety behavior from a released model.

An analogy: thousands of switches, one dial

Picture a building with thousands of light switches. It turns out that how bright a room is depends on just one particular combination of them. Build a single dial that moves along that combination and you can turn the brightness up or down. The paper's claim is that whether a chat model refuses has a dial like that, and that it is not a single neuron (one switch) but a direction spread across many of them.

Background: the residual stream and directions

In a Transformer, every token carries one vector from layer to layer, and each layer adds the output of its attention and MLP blocks onto it. The paper calls this the residual stream and writes $\mathbf{x}_i^{(l)}$ for the vector of the token at position $i$ as it enters layer $l$ (§2.1). The work rests on a widely held hypothesis: language models represent concepts as linear directions in activation space (§1).

Once you fix a direction $\hat{\mathbf{r}}$ (a vector of length 1), you can measure how much any activation $\mathbf{x}$ points that way with a single number, the dot product $\hat{\mathbf{r}}^\top\mathbf{x}$. That number is the size of the projection. Dot products are covered from scratch in Linear Algebra for AI.

Mechanism 1: finding the direction with a difference in means

The recipe is plain (§2.3). Run a set of harmful instructions and a set of harmless instructions through the model and take the average activation for each.

In words: $\boldsymbol{\mu}$ is where activations land on average when the model reads a harmful instruction, and $\boldsymbol{\nu}$ is the same for harmless ones ($\mathcal{D}$ is a set of instructions, $\mathbf{t}$ one instruction in it).

That is, the arrow from the harmless average to the harmful average. Its direction says along which axis the two groups differ; its length says how far apart their means are. This is the difference in means. Averaging is what makes it work: the quirks of individual prompts point every which way and largely cancel when you add many of them up, while whatever differs consistently between the two groups survives in the difference.

The training sets hold 128 harmful and 128 harmless instructions, with 32 of each for validation (§2.2). Each layer and each template token after the instruction gives a candidate vector, and the authors pick the one that, on validation data, removes refusal when erased, induces refusal when added, and otherwise changes behavior on harmless inputs as little as possible (§2.3, §C).

Interactive: feel the size of a projection

Treat one arrow as the refusal direction $\hat{\mathbf{r}}$ and the other as an activation $\mathbf{x}$, and rotate them. The dot product is the gauge for how much that activation rides on the refusal direction. Make them perpendicular and it drops to zero.

Mechanism 2: subtracting and adding the direction

The paper uses two interventions (§2.4). The first is directional ablation.

In words: take whatever part of $\mathbf{x}$ lies along $\hat{\mathbf{r}}$ (the projection size times the direction) and subtract all of it. The result is perpendicular to $\hat{\mathbf{r}}$, and every other component is left alone. The paper does this at every layer and every token position.