orca

Definition

A line of small language models from Microsoft Research, trained by imitating not just a larger model's final answers but its explanations and step-by-step reasoning traces. The point was that progressive, explanation-rich supervision lets a modest parameter count do surprisingly well on reasoning tasks, which is why it is cited as the canonical example of shifting distillation targets from outputs to process. The word otherwise means killer whale.