Knowledge distillation: how a big model teaches a small one

A large model is costly to run, and a small one is cheap but often knows less. Knowledge distillation is one way to narrow that gap: you train a small student model to imitate a larger teacher. Flyingkiwi’s v3.1 models were built this way, with a 35B teacher and 0.6B student models. This article explains what the teacher was asked, what was filtered, and what the results do and do not show.

The idea: a teacher’s output trains a student

Distillation starts from a simple observation. A teacher model that is large enough to answer carefully is expensive to run for every question. A student model that is small enough to run on a device is cheap, but it has less capacity and so it knows less. Distillation trains the student to behave like the teacher, using the teacher’s outputs as the training signal.

Depending on the method, the student learns from the teacher’s output probabilities or from the text the teacher wrote. The two approaches are different in what the student receives, and the difference matters for how you design the training data.

Two flavours: soft targets and written answers

The classic version, described by Hinton, Vinyals and Dean in 2015, trains the student to match the teacher’s full probability distribution over possible next tokens. Those probabilities are called soft targets. They carry information about which wrong answers were nearly right, which a single correct word does not.

Sequence-level distillation, as described by Kim and Rush in 2016, is simpler. The student is trained on sequences that the teacher produced. Flyingkiwi used this second approach. Its students learned from the teacher’s written answers, not from the teacher’s probability distributions, which is the practical reason the rules for those answers matter so much.

The teacher and its rules

The teacher was Qwen3.5-35B-A3B, an Apache-2.0 model run in FP8. It wrote answers under a written rule set. The answers had to be accurate, say when the model was unsure or when something was unknowable, correct false premises, ask when a request was ambiguous, and use Markdown and code blocks.

The rules matter because a student copies style as well as facts. If a teacher hedges without reason, invents detail or pads its answers, the student learns those habits too. The students then trained on those answers with the short system prompt that the app sends, so the training matched the way the app is actually used.

What the teacher was asked

The English set had 15,834 training and 344 validation conversations. They were drawn from several kinds of prompt:

• 3,369 human-written prompts from OASST2 and Aya;
• 653 GSM8K word problems;
• 6,292 everyday, explanation, writing, coding and maths requests written by the teacher;
• 1,890 honesty probes;
• 479 questions about invented entities;
• 125 identity questions;
• 248 memory checks;
• 1,198 chained conversations.

Of the English conversations, 1,544 continue with teacher-written follow-up turns, and 2,385 have no system prompt at all. The Turkish set had 20,778 training and 388 validation conversations. It included 7,761 human-written prompts from OASST2, Aya and translated OASST2, 433 GSM8K word problems, 7,455 teacher-written requests, 1,941 honesty probes, 460 invented-entity questions, 118 identity questions, 474 memory checks and 1,567 chained conversations.

The honesty probes teach the student to admit uncertainty when a question has no reliable answer. Invented-entity questions ask about things that do not exist, so the honest reply is to say that the thing is unknown rather than to describe it. Memory checks and chained conversations train the student to use earlier turns in the same chat. Each category targets a behaviour that a small model is likely to get wrong if nobody trains it.

Calibration from the student’s own knowledge

Some training examples teach the model to say when it does not know. These were built from TriviaQA questions. The base model, Qwen3-0.6B, was asked each question, and the questions were sorted by whether it could answer them. Known answers were kept, and unknown ones were taught with an honest “I’m not sure”.

The reason to build calibration from the student’s own knowledge is that the uncertainty to teach depends on what the student actually knows. A teacher with more knowledge would mark many questions as answerable, even though the small model cannot answer them. The student would then learn to answer with confidence where it has no support.

Quality control: filtering and the concise rewrite

GSM8K gold answers were used to filter the teacher’s answers to maths problems. Cleaning for v3 filtered or reordered answers that contained unprompted remarks about the assistant’s own limits, thinking aloud, or maths results stated before the working.

The concise rewrite was new in v3.1. The v3 answers were two to four times as long as the base model’s, and blind grading flagged more invented details. Asking for brevity in the system prompt did not shorten them. So the teacher saw each whole conversation and rewrote every assistant message into the shortest complete answer.

The rewrite had to keep code blocks, requested formats, honest statements of uncertainty, and anything a later turn refers to. A rewrite was rejected if it dropped a code block, added a disclaimer, grew longer, or switched language. In English, 22,067 answers were rewritten and 43% of the characters were kept. In Turkish, 30,036 were rewritten and 33% were kept. The reasoning is that every extra sentence is another chance for a 0.6B model to state something false, as the article on invented details describes.

The recipe and the compute

Training was full-parameter fine-tuning in BF16 with FP32 master weights, for two epochs. The peak learning rate was 1.5e-05 with cosine decay. No quantisation-aware training was used for these models.

English took 148 steps, and its validation loss fell from 2.165 to 1.311. Turkish took 300 steps, and its validation loss fell from 2.610 to 1.632. These are the training run’s own validation losses, so they are not the same measurement as the answer-loss table in the quantisation article.

The compute was one NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB, in an existing Colab Pro+ session. Rewriting took 57 minutes. Training took 7 minutes for English and 15 minutes for Turkish. The whole Colab session lasted about two hours.

What the results show, and what they do not

The student is limited by what it can hold. A 0.6B model cannot store everything a 35B teacher knows, and it copies the teacher’s mistakes along with the good answers. Distillation does not add a teacher’s knowledge for free.

On the frozen test sets, English scored 63 out of 100, the same as the unchanged Qwen3-0.6B. Turkish scored 37 out of 100, against 12 for the base model. English also corrected false premises less often, at 5 out of 10 against 7 for the base model. Turkish quality was judged by an AI grader rather than by native speakers, and 37 out of 100 is low in absolute terms. The article on what a 0.6B model can do covers the failures in more detail.

The reasonable reading is that distillation mostly changed behaviour, such as honesty, brevity and language, rather than adding knowledge. English v3.1 said it could not know something 18 times out of 20, against 17 for the base model. Turkish v3.1 said so 14 times out of 20, against 1. On answerable facts and reasoning, English v3.1 scored 20 out of 30 against the base model’s 22, so no knowledge was visibly added. This is an interpretation of the numbers, not a proven fact, because the experiment did not isolate which training data caused each change. The technical report lists the training details.