knowledge

Definition

The factual and relational content a model carries, either baked into weights during training (parametric) or supplied at run time by retrieval (non-parametric). The distinction organizes much practical work: parametric knowledge is fast but frozen at the training cutoff and hard to audit or update, while retrieved knowledge is current and citable but only as good as the retriever.

Explainers using this term

• Distillation vs. Quantization vs. Pruning — Three Roads to a Smaller ModelDistilling the Knowledge in a Neural Network
• A Field Guide to Distillation Recipes — logit, feature, attention, selfDistilling the Knowledge in a Neural Network
• The Math of Distillation — Why Soft Answers Teach MoreDistilling the Knowledge in a Neural Network