context

Definition

The span of input a model can attend to in a single forward pass, measured in tokens. Anything outside it effectively does not exist for the model, which is why fitting long documents, chat history, and retrieved passages into the window is a central engineering problem. The key distinction is against parametric knowledge: context is supplied fresh at inference time and disappears afterwards, whereas weights carry what was learned during training.

Explainers using this term

• Paper Walkthrough: SAS — Training a Sparse-Attention Block Selector End-to-End With the Language Modeling LossSAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
• Paper Walkthrough: Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training — training the draft alongside a 122B policy at 256K tokensOnline Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
• Paper Walkthrough — The Missing Temporal Link: Putting Script Timing on the Video–Audio Axis with Temporal Context RoutingThe Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation