How distillation actually works

Diogo AlmeidaDealroom has a profile for this one. Try Dealroom → — ex-OpenAI researcher, RLHF/InstructGPT co-author, and Co-Founder and CEO of TypeSafe AIDealroom has a profile for this one. Try Dealroom → — published a long-form explainer in July 2026 cutting through the row over whether Moonshot AI's Kimi K3 distilled Anthropic's models. His argument: the term "distillation" is being used loosely, and the reality is more nuanced than the framing suggests.

Three types of distillation. He separates them by how much information transfers: logit distillation (copying the teacher's raw token probabilities — the original meaning, now impossible cross-lab because frontier APIs no longer return logits); behaviour cloning (copying the teacher's output tokens via supervised fine-tuning — also blocked today, since reasoning models return only summarised traces, not their real chain-of-thought); and what he calls behaviour parroting (mimicking the final outputs of hidden reasoning work, without ever seeing the reasoning).

Only parroting is left cross-lab. With logits and full reasoning traces withheld, behaviour parroting is the one route still available to a lab training on a rival's commercial API.

Does parroting actually work? Intuitively it should not — copying outputs of secret work ought to produce plausible-looking hallucination. But he argues it can, on three grounds: much of instruction-following, and even reasoning, is style (training on outputs alone closes most of the gap); reasoning models train nearly as well on traces that lead to wrong answers, implying the gains are stylistic rather than logical; and large-scale pre-training is itself a form of behaviour parroting on internet text, which plainly works.

Bottom line. Distillation from commercial APIs is plausible, but it is "an approximation (parroting) of an approximation (cloning) of the real thing (logits)" — and, in his view, both the debate and the US government's claims conflate these distinct techniques.

Read more: X (@CompleteSkeptic) · Complete Skeptic