Paper Explained: Negative Self-Distillation — Learning to Reason by Steering Away From Your Worst Self
Self-distillation from ground-truth solutions (OPSD) can strip a reasoning model of its habit of pausing and double-checking. NSD has the model play a 'careless reasoner' and trains only to move away from that bad teacher — no labels required. A from-scratch walkthrough.
Key points
- NSD trains the model to diverge from flawed reasoning instead of imitating privileged solutions.
- With no ground truth or external supervision, the model writes a question-specific negative condition for itself, and the student's distribution is pushed away from this self-made negative teacher.
- OPSD has the student imitate an artificially confident trace from a teacher that knows the answer, which suppresses expressions of uncertainty and self-correcting behavior.
- Because flawed reasoning tokens are tangled up with ordinary language tokens, NSD uses a dynamic gate to isolate reasoning-critical tokens.
- The authors report that NSD consistently beats OPSD and other label-free, self-bootstrapping RL baselines.
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Primary source — what this article is built on
undefined2026-09-10→undefined2026-09-25same month
Negative Self-Distillation: Learning to Reason by Avoiding FlawsRongcan Pei, Zhepei Wei, Shuyao Xu et al. · 2026-09-10 · v1arXiv:2609.11699Paper page·PDFundefined
On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.
What the paper claims
This article covers "Negative Self-Distillation: Learning to Reason by Avoiding Flaws" (Pei, Wei, Xu, Zhu, Chen, Meng; arXiv:2609.11699, 2026-09-10), or NSD for short.
The abstract, in brief: On-Policy Self-Distillation (OPSD) lets an LLM act as its own teacher by giving that teacher privileged information such as the reference solution, and it has become a popular way to self-improve. But recent work shows OPSD can badly hurt performance on hard reasoning tasks. Imitating an artificially confident trace produced by a teacher that already knows the answer suppresses expressions of uncertainty and penalizes the exploratory, self-correcting behavior that difficult problems require. NSD flips the direction: instead of imitating privileged solutions, it trains the model to diverge from flawed reasoning. With no ground truth or external supervision, the model writes a question-specific "negative condition" for itself (e.g., "act as a careless reasoner"), and the student's distribution is pushed away from this self-made negative teacher. Naively applying an unlearning objective would be dangerous, because flawed reasoning tokens are tangled up with ordinary language tokens, and punishing both could wreck the model's basic language ability. NSD solves this with a dynamic gate that isolates reasoning-critical tokens, and the authors report that it consistently beats OPSD and other label-free, self-bootstrapping RL baselines.
An analogy: copying the answer key vs. avoiding the bad example
Picture two math tutors.
- The OPSD tutor explains every problem while already holding the answer. The walkthrough is a straight line with zero hesitation. A student who copies it gradually loses the habit of stopping to say "wait, is that right?" — exactly the habit they need on an exam where nobody hands them the answer.
- The NSD tutor does the opposite: "Here's what a sloppy solution looks like. Don't go that way." It never shows the right path, but once you know where the pitfalls are, you can find the rest yourself.
The twist is that the sloppy example is acted out by the student itself. No external teacher, no labeled data.
Background: why imitating the right answer can make models weaker
Following §1: the standard way to boost reasoning is reinforcement learning with verifiable rewards (RLVR). It is expensive (several rollouts per question), the signal vanishes when a whole group is all-right or all-wrong, and a single outcome reward smeared over the entire answer can't say which token mattered.
On-policy distillation fixes the last problem by having a teacher score every token of the student's own rollout (see our on-policy distillation explainer). OPSD removes the need for a stronger external teacher by letting the same model play teacher after seeing the reference solution.
The catch, per the paper, is that a teacher who has seen the answer tends to be overconfident and linear. Pulling the student toward that distribution suppresses high-entropy exploration, hedging, and self-correction (§1).
Drag the temperature down in the widget below: the distribution collapses onto a single option and everything else shrinks toward zero. That is roughly what happens to minority tokens like "Wait" when a student imitates an answer-aware teacher.
Mechanism 1: build your own "bad teacher"
NSD starts from an unlabeled problem set , where is a problem statement. Under the default "online" strategy, each problem goes through this step (§2.1):
Comments
Sign in to comment