off-policy

Definition

Learning a target policy from experience generated by some other policy — old data, a replay buffer, a human demonstrator, or an exploratory behavior policy. Reusing data makes off-policy methods far more sample-efficient, which is why Q-learning, DQN, and SAC are built this way. The contrast is on-policy learning, such as PPO, where every update must use fresh data from the current policy; off-policy buys efficiency at the cost of distribution mismatch that the algorithm has to correct for.

Explainers using this term

• Paper Walkthrough — WarpSAC: When RL's Safety Rails Become HandcuffsWarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation