imitating

Definition

Training one model on another model's outputs so that it reproduces their style and format. The critique made by work on imitating proprietary LLMs is that surface imitation is cheap while the underlying capability gap barely narrows: the student learns to sound like the teacher, and human raters are fooled before benchmarks are. Related to distillation, but typically without access to the teacher's internals.

Explainers using this term

• Evaluating Distilled Models — Is "Close to the Teacher" a Good Metric?The False Promise of Imitating Proprietary LLMs