Inside Blazing Kiwi: a 70 MB model that searches its own library

Blazing Kiwi is Flyingkiwi’s new default English model: 122 million parameters, a 69.5 MB file and an offline library of 49,690 Wikipedia summaries that it searches with keys made from its own hidden state. This article walks through the engineering behind it, with the arithmetic that set each limit. The full method and every measurement are in the technical paper; the non-technical introduction tells the same story with analogies.

The budget decided the vocabulary, not the depth

The starting point is SmolLM2-135M-Instruct, an Apache-2.0 model from Hugging Face: 30 Llama-style layers of width 576, 9 query heads and 3 key-value heads, a feed-forward block of 1,536 and an embedding tied to the output layer. Each layer holds 3,538,944 weights. Stored as 4-bit codes with one 16-bit scale per 32 weights, every weight costs 4.5 bits, so the 30 layers alone take 59.7 MB. Whatever is left of the 70 MB target goes to the embedding, which has one row of 576 weights per vocabulary entry: 324 bytes a row.

SmolLM2 has 49,152 entries, which would cost 15.9 MB and push the file to 76.4 MB. Two ways out were open: drop layers or drop vocabulary. Layers are used by every token; most of the vocabulary serves code and other languages and is rarely used in English chat. So the vocabulary was cut. Every token used at least 37 times across 19.2 million tokens of training conversations and library text was kept, plus every single-byte token the base vocabulary has and Latin letters, and the set was closed over BPE merge ancestors, so text made of kept tokens is split exactly as before. That leaves 27,799 entries and a 69.5 MB file; English text needs 1.97% more tokens than with the full vocabulary.

Flyingkiwi’s 0.6B models stay in 8 bits because rounding a model trained in 16 bits to 4 bits cost them accuracy. Blazing Kiwi avoids that by training in 4 bits: for the last 30% of steps every matrix, the embedding included, runs through the engine’s exact quantiser while gradients pass straight through the rounding. The codes in the file are the ones the model was trained with. It is not free: after 4-bit training the cross-entropy on held-out answers is 1.549 nats per token, against 1.464 for the full-precision model where 4-bit training began, a gap of about 0.085 nats.

A search key made by the model itself

A separate embedding model would not fit, so the language model produces its own key. While the engine reads the user’s message, it takes the residual stream after layer 24 for each of the message’s tokens, divides each vector by its root-mean-square, and averages them into one 576-dimensional vector h. A learned 256 × 576 matrix projects h to 256 numbers, and their signs are the key: 256 bits, 32 bytes. Nothing extra is computed for the state itself, because that layer runs anyway on the way to the reply; the projection is 147,456 multiplications.

Library entries get their keys ahead of time from the same layer, through their own projection matrix, so a question and the passage that answers it can be mapped to nearby keys even though they look different. Only the query projection ships. Both projections were trained with a contrastive loss on 17,096 pairs of a question and its passage, with the codes relaxed to tanh(βz) and β rising from 1 to 6 so that training ends close to the signs used at run time. Eight candidate layers were tried; layer 24, six from the top, worked best, putting the right passage among the first five for 53.0% of held-out questions against 45.2% for the next best, layer 27.

Searching 49,690 entries in a millisecond

The key turned out to be the weakest part of the search. On its own it puts the right passage for a held-out question first only 29.7% of the time, and among the first five 57.5%: a 256-bit summary of a 122M model’s state does not separate 49,690 encyclopedia topics. So every entry is scored with three terms:

score = BM25(message, entry) + 32 × (agree − differ) / 256 + title bonus

Read as vectors of +1 and −1, two keys have a dot product equal to the number of agreeing bits minus the number of differing bits, 256 − 2 × popcount(query XOR entry): four 64-bit popcnt instructions per entry. BM25 is the classic search-engine formula, computed here on token ids instead of words. Because the passages are stored as the model’s own token ids, the library controller builds the inverted index, 3.28 million postings, when the library loads (about 0.12 s), so no index is downloaded. Question words such as “what” or “explain” are rare in encyclopedia text and would otherwise count heavily, so 440 of their token forms weigh nothing in queries. Byte-level BPE also gives “Chlorophyll”, “ Chlorophyll” and “ chlorophyll” different ids, so each title is indexed in all three forms. The title bonus adds 8 points when a title of two or more words appears in the message and 4 for a one-word title. These are version 1.1’s weights; the first version, with different keys, used 64 for the key term and 16, 10 and 4 for the titles.

The weights were chosen on half of two held-out question sets and measured on the other half. For questions written from passages that training never saw, the right passage was among the two notes 94.1% of the time, against 87.8% for word-level BM25 over the same library; for short questions that name a title, such as “What is chlorophyll?”, it was 97.5%. A full scan takes 0.27 ms (median) in single-threaded WebAssembly.

The library file stores each passage as token ids of the model’s own vocabulary (6.0 MB after gzip, against 7.5 MB as compressed text), titles and short descriptions as text, the keys, the BM25 tables and a title index. It is 10.6 MB as downloaded and is searched by an 8 KB WebAssembly module written in freestanding C. Nothing about the search leaves the device: the whole library is downloaded once, like the model.

Reading notes without being misled by them

The best two entries become notes in a separate library turn after the user’s message; the next three are listed by title and description. If the best score is below a gate, no notes are added. Retrieval is imperfect, so the model has to learn when notes help. The training data was built for that. Qwen3.5-35B-A3B answered 9,856 questions about library passages with two notes, the right passage and a near neighbour, in random order, and in 12% of them the right passage was left out on purpose; half of 15,834 general conversations got the top two BM25 notes whether they were relevant or not, and the teacher answered again with them in view. Answers that mentioned “the notes”, or cited a page that was not offered, were dropped.

The student learned from two signals at once. Half of the loss is the usual cross-entropy on the teacher’s written answer. The other half is the KL divergence to a 1.7B model of the same family, SmolLM2-1.7B-Instruct, first fine-tuned on the same 30,209 conversations. Because the two share a tokenizer, the student sees the full next-token distribution over its 27,799 tokens, not just the token that was written. Fine-tuning brought the assistant’s cross-entropy on held-out answers from 1.517 to 1.083 nats per token; the student’s fell from 1.985 to 1.464 before 4-bit training, and its KL divergence to the assistant from 0.933 to 0.533.

Citations as a grammar

Every answer that uses a note ends with Read more: [[Title]], which the page renders as a link. A model this small would sometimes invent a title, so the decoder does not let it. Before each token, the library controller runs the bytes of every token that contains a bracket or a backtick through a small automaton. Outside a citation everything is allowed. After [[, a token is allowed only if the bytes written so far stay a prefix of one of the five offered titles (or their Wikivoyage pages), and ]] only after a complete title. With no library turn, [[ cannot be written at all. Code blocks are left alone. The same mechanism already keeps Flyingkiwi’s answer formats inside their templates.

What it costs and what it buys

The download fell from 571 MB to 80.9 MB: 69.5 MB of weights, a 10.6 MB library, the tokenizer and the engines. In Chrome on a laptop processor, Blazing Kiwi wrote 98 to 134 tokens per second against 29 to 35 for the 0.6B model in the same test, and the largest Chrome process peaked at 509 MB against 1,237 MB. On 200 questions about held-out library passages, graded blind, version 1.1 answered 120 correctly, against 25 for the 0.6B model and 103 for version 1.0 in the same grading. In the first version’s tests the library switched off gave 8, and switching off only the copy bonus, a logit bonus of 2 on the notes’ own tokens, gave 58 against 100. Against PyTorch running the same 4-bit codes, the WebAssembly engine’s next-token distribution differs by a mean KL divergence of about 0.001, and its recall keys agree on 254 of 256 bits.

The price is general ability. On Flyingkiwi’s frozen set of 100 cases, which tests arithmetic, short logic, false premises, ambiguity and conversation rather than knowledge, Blazing Kiwi 1.1 scores 25 against 64 for unchanged Qwen3-0.6B and 66 for Flyingkiwi’s 0.6B model, and it invents details more often (20 flagged answers against 4). Version 1.1 added 9,418 conversations aimed at exactly these behaviours and scored the same as version 1.0, 25 and 25: the limit looks like the model’s size, not its lessons. Larger models of the same family, unchanged and graded with it, score 30 (360M parameters, about 204 MB in 4 bits) and 73 (1.7B, about 963 MB): at 70 MB the number of weights sets the ceiling. What the library adds is knowledge, and there it beats the 0.6B model clearly; what it cannot add is reasoning. A library cannot make a 122M model reason. Even with the right note it answered only about half of the library questions fully correctly; the usual error is a wrong date, work or place added to an otherwise correct summary. It works in English only, and its library holds the opening of each of 49,690 articles as of 30 October 2026. The 0.6B model stays one click away in the model menu.