Its main idea is simple. A model this small cannot hold much knowledge in its own numbers, so Blazing Kiwi is paired with a library it can search. It is taught to read what it finds and to say where it came from. This article explains how that works, with pictures rather than equations.

A small student with a good index

Picture two students sitting the same exam. The first has tried to memorise an entire encyclopedia. The second carries a small pocket library with a good index, and checks the right page before answering. The first student knows more by heart, but only up to the limit of memory. The second knows less by heart, but can find what the question needs.

Blazing Kiwi is built like the second student. Its library is a set of short article summaries, downloaded with the model and searched on your device. The model reads the notes it finds and uses them when they help.

Making a small model smaller still

The model file is 69.5 MB, under the 70 MB target. Two changes make that possible. The first is the number format. Every number inside the model is stored in 4 bits, with one shared scale for each group of 32 numbers. Think of a shop that rounds every price to the nearest coin. The small change is lost, but the price still tells you roughly what something costs. The training described below also practises with rounded numbers, so the model learns to work with them.

The second change is the vocabulary. A model reads text in word pieces, and its list of pieces was cut from 49,152 to 27,799. The pieces that remain are the ones English chat and its library actually use. It is like a dictionary that keeps only the words you use every day.

For comparison, the 0.6B model is a 571 MB download. Measured in the same browser on the same laptop, its tab peaked at 1,237 MB of memory and Blazing Kiwi’s at 509 MB, and Blazing Kiwi wrote its replies about three and a half times faster.

The library it reads from

The library holds the opening sentences of 49,690 English Wikipedia articles, each limited to about 110 word pieces. The articles come from Wikipedia’s “vital articles” lists at Levels 4 and 5, which editors curate to cover all the major subjects. The summaries were fetched on 30 October 2026. For 3,270 of them, places with a travel guide, the entry also points to Wikivoyage.

The library adds 10.6 MB to the model’s download. Its text is Wikipedia’s own, licensed under CC BY-SA 4.0, which asks for credit to the authors and the same licence for adapted text. This article gives that credit: the summaries come from Wikipedia and its contributors.

Keys and matching

To find the right article, the app needs a fast way to compare your question with 49,690 summaries. It uses a 256-bit key. While the model reads your message, one of its internal layers is averaged over your words. The result is turned into 256 yes or no answers, one for each of 256 numbers, depending on whether that number is positive or negative. Every library entry has its own key, worked out in advance, so at question time only your message needs one.

The picture for this is an old paper filing system: edge-notched index cards, known as McBee Keysort cards, which were used in the middle of the twentieth century. Each card has a row of holes along its edge, and a notch can be cut at any hole or left whole. A card is therefore a row of yes and no answers. A needle pushed through a hole lifts out the cards that are notched there.

Blazing Kiwi does something similar, with a score instead of a needle. It compares your key with all 49,690 keys, one position at a time: each position where the two keys agree adds a point, and each position where they differ takes one away. On its own, the key is only a rough guide, so the score adds two simpler clues. The first is word matching, the way a classic search engine works: word pieces that your message shares with an entry count, and rare pieces count far more than common ones, so “chlorophyll” matters and “what” does not. The second is a bonus when an article’s title appears in your message, so a question that names its subject finds its article. The entry with the highest total is the closest match.

How a question is answered

Here is the path for one message. First, the model reads your words and makes the key. Second, the app scores every library entry with the key, the word pieces of your message and the titles, and picks the best matches. The whole search takes about a quarter of a millisecond on the device. The two best entries are given to the model as notes it reads before answering. The notes sit in a separate part of the conversation, so you do not see them. The next three entries are listed by title only. While the model writes, the app gives a small nudge to every word piece that appears in the two notes, so the model leans on the article’s own wording instead of improvising. In tests of the first version, that nudge alone raised the number of correct answers to 200 library questions from 58 to 100.

If even the best match is weak, no notes are added, and the model answers from what it has learned, which is limited. The model was trained to use a note when it answers, to ignore notes that do not help, and to finish with a line reading “Read more” followed by a title in double square brackets, whenever it used one. The page turns that line into a link to the Wikipedia article. On held-back test questions written from passages the model never trained on, the right passage was among the two notes 94.1% of the time; for short questions that name a subject, such as “What is chlorophyll?”, it was 97.5%.

A librarian who can only recommend what is on the shelf

Think of a librarian who may only recommend books that are on the shelf in front of them. Blazing Kiwi works in the same way when it cites a source. A small program in the decoder, the part that writes the reply one piece at a time, applies a rule here. After an opening double bracket, the model may only write one of the five titles it was offered.

So every link points to a page the model was actually shown. This makes each link checkable, but it does not make the sentence before it correct. The model can still misread a note, so open the linked article and check what it says.

How it was trained

A much larger open model, Qwen3.5-35B-A3B, wrote about 30,000 example conversations. They covered questions about library passages answered from notes, general conversation, answers in five layouts (fact cards, flowcharts, diagrams, tables, and pros and cons), questions asking where to read more about a subject, questions about the model itself, and follow-up questions.

Training then happened in two steps. A mid-sized model from the same family, SmolLM2-1.7B-Instruct, was first trained on those conversations. Blazing Kiwi, which is built from SmolLM2-135M-Instruct, an open model by Hugging Face, learned from two sources. One was the written answers. The other was the mid-sized model’s word-by-word probabilities, which work as a teacher assistant. For the last 30% of training, it practised with its numbers already rounded to 4 bits. The file you download is therefore the model that was trained.

Of the library passages, 600 were held back entirely for testing, so the test passages were never used in training. Training ran on one rented GPU in Google Colab.

What it is bad at

Blazing Kiwi has clear limits. Its arithmetic is weak, and so is its multi-step reasoning. When the library has nothing relevant, it knows little. It can misread a note or mix up two notes, and it works in English only. Do not use it for sums you need to be exact.

For harder questions, the 0.6B model is one click away in the model menu. It stays available, and the Turkish model is unchanged. On Flyingkiwi’s frozen test set of 100 reasoning and conversation cases, graded blind, Blazing Kiwi scored 25, against 64 for the unchanged 0.6B base model and 66 for Flyingkiwi’s own 0.6B model. On 200 questions about library articles it never trained on, it answered 120 correctly, against 25 for the 0.6B model, which has no library. The current version, 1.1, learned from 9,400 more example conversations about false premises, unclear requests and other everyday situations. It answered more library questions correctly than the first version (120 against 103) but scored the same on reasoning, which suggests the limit is the model’s size rather than its lessons. On the same test, a model of the same family about three times the size scored 30 without any of this training, and one about twelve times the size scored 73.

Use it for short, plain questions about the subjects in its library, for simple wording and for explaining ideas. Then check the answer against the linked article, especially names, dates and figures.

The budget decided the vocabulary, not the depth

The starting point is SmolLM2-135M-Instruct, an Apache-2.0 model from Hugging Face: 30 Llama-style layers of width 576, 9 query heads and 3 key-value heads, a feed-forward block of 1,536 and an embedding tied to the output layer. Each layer holds 3,538,944 weights. Stored as 4-bit codes with one 16-bit scale per 32 weights, every weight costs 4.5 bits, so the 30 layers alone take 59.7 MB. Whatever is left of the 70 MB target goes to the embedding, which has one row of 576 weights per vocabulary entry: 324 bytes a row.

SmolLM2 has 49,152 entries, which would cost 15.9 MB and push the file to 76.4 MB. Two ways out were open: drop layers or drop vocabulary. Layers are used by every token; most of the vocabulary serves code and other languages and is rarely used in English chat. So the vocabulary was cut. Every token used at least 37 times across 19.2 million tokens of training conversations and library text was kept, plus every single-byte token the base vocabulary has and Latin letters, and the set was closed over BPE merge ancestors, so text made of kept tokens is split exactly as before. That leaves 27,799 entries and a 69.5 MB file; English text needs 1.97% more tokens than with the full vocabulary.

Flyingkiwi’s 0.6B models stay in 8 bits because rounding a model trained in 16 bits to 4 bits cost them accuracy. Blazing Kiwi avoids that by training in 4 bits: for the last 30% of steps every matrix, the embedding included, runs through the engine’s exact quantiser while gradients pass straight through the rounding. The codes in the file are the ones the model was trained with. It is not free: after 4-bit training the cross-entropy on held-out answers is 1.549 nats per token, against 1.464 for the full-precision model where 4-bit training began, a gap of about 0.085 nats.

A search key made by the model itself

A separate embedding model would not fit, so the language model produces its own key. While the engine reads the user’s message, it takes the residual stream after layer 24 for each of the message’s tokens, divides each vector by its root-mean-square, and averages them into one 576-dimensional vector h. A learned 256 × 576 matrix projects h to 256 numbers, and their signs are the key: 256 bits, 32 bytes. Nothing extra is computed for the state itself, because that layer runs anyway on the way to the reply; the projection is 147,456 multiplications.

Library entries get their keys ahead of time from the same layer, through their own projection matrix, so a question and the passage that answers it can be mapped to nearby keys even though they look different. Only the query projection ships. Both projections were trained with a contrastive loss on 17,096 pairs of a question and its passage, with the codes relaxed to tanh(βz) and β rising from 1 to 6 so that training ends close to the signs used at run time. Eight candidate layers were tried; layer 24, six from the top, worked best, putting the right passage among the first five for 53.0% of held-out questions against 45.2% for the next best, layer 27.

Searching 49,690 entries in a millisecond

The key turned out to be the weakest part of the search. On its own it puts the right passage for a held-out question first only 29.7% of the time, and among the first five 57.5%: a 256-bit summary of a 122M model’s state does not separate 49,690 encyclopedia topics. So every entry is scored with three terms:

score = BM25(message, entry) + 32 × (agree − differ) / 256 + title bonus

Read as vectors of +1 and −1, two keys have a dot product equal to the number of agreeing bits minus the number of differing bits, 256 − 2 × popcount(query XOR entry): four 64-bit popcnt instructions per entry. BM25 is the classic search-engine formula, computed here on token ids instead of words. Because the passages are stored as the model’s own token ids, the library controller builds the inverted index, 3.28 million postings, when the library loads (about 0.12 s), so no index is downloaded. Question words such as “what” or “explain” are rare in encyclopedia text and would otherwise count heavily, so 440 of their token forms weigh nothing in queries. Byte-level BPE also gives “Chlorophyll”, “ Chlorophyll” and “ chlorophyll” different ids, so each title is indexed in all three forms. The title bonus adds 8 points when a title of two or more words appears in the message and 4 for a one-word title. These are version 1.1’s weights; the first version, with different keys, used 64 for the key term and 16, 10 and 4 for the titles.

The weights were chosen on half of two held-out question sets and measured on the other half. For questions written from passages that training never saw, the right passage was among the two notes 94.1% of the time, against 87.8% for word-level BM25 over the same library; for short questions that name a title, such as “What is chlorophyll?”, it was 97.5%. A full scan takes 0.27 ms (median) in single-threaded WebAssembly.

The library file stores each passage as token ids of the model’s own vocabulary (6.0 MB after gzip, against 7.5 MB as compressed text), titles and short descriptions as text, the keys, the BM25 tables and a title index. It is 10.6 MB as downloaded and is searched by an 8 KB WebAssembly module written in freestanding C. Nothing about the search leaves the device: the whole library is downloaded once, like the model.

Reading notes without being misled by them

The best two entries become notes in a separate library turn after the user’s message; the next three are listed by title and description. If the best score is below a gate, no notes are added. Retrieval is imperfect, so the model has to learn when notes help. The training data was built for that. Qwen3.5-35B-A3B answered 9,856 questions about library passages with two notes, the right passage and a near neighbour, in random order, and in 12% of them the right passage was left out on purpose; half of 15,834 general conversations got the top two BM25 notes whether they were relevant or not, and the teacher answered again with them in view. Answers that mentioned “the notes”, or cited a page that was not offered, were dropped.

The student learned from two signals at once. Half of the loss is the usual cross-entropy on the teacher’s written answer. The other half is the KL divergence to a 1.7B model of the same family, SmolLM2-1.7B-Instruct, first fine-tuned on the same 30,209 conversations. Because the two share a tokenizer, the student sees the full next-token distribution over its 27,799 tokens, not just the token that was written. Fine-tuning brought the assistant’s cross-entropy on held-out answers from 1.517 to 1.083 nats per token; the student’s fell from 1.985 to 1.464 before 4-bit training, and its KL divergence to the assistant from 0.933 to 0.533.

Citations as a grammar

Every answer that uses a note ends with Read more: [[Title]], which the page renders as a link. A model this small would sometimes invent a title, so the decoder does not let it. Before each token, the library controller runs the bytes of every token that contains a bracket or a backtick through a small automaton. Outside a citation everything is allowed. After [[, a token is allowed only if the bytes written so far stay a prefix of one of the five offered titles (or their Wikivoyage pages), and ]] only after a complete title. With no library turn, [[ cannot be written at all. Code blocks are left alone. The same mechanism already keeps Flyingkiwi’s answer formats inside their templates.

What it costs and what it buys

The download fell from 571 MB to 80.9 MB: 69.5 MB of weights, a 10.6 MB library, the tokenizer and the engines. In Chrome on a laptop processor, Blazing Kiwi wrote 98 to 134 tokens per second against 29 to 35 for the 0.6B model in the same test, and the largest Chrome process peaked at 509 MB against 1,237 MB. On 200 questions about held-out library passages, graded blind, version 1.1 answered 120 correctly, against 25 for the 0.6B model and 103 for version 1.0 in the same grading. In the first version’s tests the library switched off gave 8, and switching off only the copy bonus, a logit bonus of 2 on the notes’ own tokens, gave 58 against 100. Against PyTorch running the same 4-bit codes, the WebAssembly engine’s next-token distribution differs by a mean KL divergence of about 0.001, and its recall keys agree on 254 of 256 bits.

The price is general ability. On Flyingkiwi’s frozen set of 100 cases, which tests arithmetic, short logic, false premises, ambiguity and conversation rather than knowledge, Blazing Kiwi 1.1 scores 25 against 64 for unchanged Qwen3-0.6B and 66 for Flyingkiwi’s 0.6B model, and it invents details more often (20 flagged answers against 4). Version 1.1 added 9,418 conversations aimed at exactly these behaviours and scored the same as version 1.0, 25 and 25: the limit looks like the model’s size, not its lessons. Larger models of the same family, unchanged and graded with it, score 30 (360M parameters, about 204 MB in 4 bits) and 73 (1.7B, about 963 MB): at 70 MB the number of weights sets the ceiling. What the library adds is knowledge, and there it beats the 0.6B model clearly; what it cannot add is reasoning. A library cannot make a 122M model reason. Even with the right note it answered only about half of the library questions fully correctly; the usual error is a wrong date, work or place added to an otherwise correct summary. It works in English only, and its library holds the opening of each of 49,690 articles as of 30 October 2026. The 0.6B model stays one click away in the model menu.

What “making things up” means

In this sense, a hallucination is a fluent statement that the model has no good support for. The model has no intention to deceive. It produces a sentence that fits the pattern of an answer, and nothing in that process checks the sentence against a source.

A plain mistake and an invented detail are different problems, although they often appear together. An invented detail is a claim with no support behind it, and it can sound as confident as a claim that happens to be true. That is why the useful question is not only “is this right?” but “what would show that it is right?”

Here is an illustration, not a quotation from any model. You ask about a local festival in a town you name. The reply gives a date, a start time and the name of an organiser. Each detail looks plausible, and any of them could be invented. Only a check against the town’s own calendar would show which, if any, are real.

Why it happens

A language model predicts which token is likely to come next, based on the text it learned from. It does not look up a fact, check it and then write it down. Most of the time the likely continuation and the true one are the same, which is why the output is useful. When they differ, the model can still produce a smooth continuation.

Small models add two pressures. A 0.6B-class model, with 528 million parameters in English after vocabulary trimming, stores much less than a large assistant, so it has fewer facts to draw on and more gaps to fill. Training can add a third. If a model learns from answers that people judged helpful, it may learn that an answer is always expected. This is a common explanation. Flyingkiwi’s results fit it, but they do not prove it.

Every extra sentence is another chance to be wrong. A short answer with one checkable claim gives you one thing to verify. A long answer may contain many claims, and you are likely to check only the first few.

What Flyingkiwi measured in v3

Flyingkiwi’s v3 release was fine-tuned from Qwen3-0.6B on answers written by a larger teacher model. The English release blended that fine-tune 50/50 with the base model, and the Turkish release used the fine-tune alone. Students learn the style of their teacher as well as its facts, and v3 learned to write long. On the frozen test sets, which are 100 project-written cases per language that were never used for training, v3 answers averaged 83 tokens in English against 37 for the unchanged base model. In Turkish the gap was wider: 256 tokens against 74.

The blind grading also counted replies that contained invented facts or claims. In English, v3 had 11 such replies out of 100, against 6 for the base model in the same grading run; in Turkish, 28 against 26. On TruthfulQA, from Lin, Hilton and Evans, a multiple-choice test built from questions on which some people would answer falsely because of a common misconception, v3 scored 21.3% against the base model’s 25.8%. That gap is larger than the test’s sampling error of about 1.5 points.

A brevity instruction did not fix it

The first idea was the obvious one: ask for short answers in the system prompt. It did not work, and the answers stayed long. One plausible reading is that the length had been learned from the training answers themselves, so a line of instruction could not undo it. That is an interpretation rather than a proven cause, but it explains why the fix had to change the training data.

Rewriting the training answers

For v3.1, the teacher saw each whole conversation and rewrote every assistant message into the shortest complete answer. The rules were strict. Code blocks, requested formats, honest statements of uncertainty and anything a later turn refers to had to survive. A rewrite that dropped a code block, added a disclaimer, grew longer or switched language was rejected, and the original answer was kept.

Whole conversations were used because a later turn can depend on an earlier answer. A follow-up question may say “the second option” or repeat a name from before. Shortening an earlier answer had to leave those references intact, or the later turn would no longer make sense.

The rewrite covered 22,067 English answers, keeping 43% of their characters, and 30,036 Turkish answers, keeping 33%. The reasoning was simple. Every extra sentence is another chance for a 0.6B-class model to state something false, so the training answers should contain only sentences that do work.

What the results show

The table compares v3 and v3.1 with the unchanged base model, Qwen3-0.6B, on the same frozen sets. The base-model column comes from the v3.1 grading run; in v3’s own run the base model had 6 English and 26 Turkish invented-detail replies.

On English, v3.1 matches the base model on invented details, and it gets the same 63 of 100 cases overall. The rewrite removed a difference on this test, not every mistake. In Turkish, v3.1 has fewer invented-detail replies than the base model, 17 against 19, and its answers are still longer than the base model’s. Its score of 37 of 100, against 12 for the base model, is a large gain, and it is still not a reliable assistant.

Read these numbers with their limits. The grading was done by AI graders against reference answers, with spot checks, and there was no independent human or native-speaker panel. Each case moves a category score by 3–10 points. The flags count replies, not individual false claims, so a reply with several false claims counts once. English v3.1 also corrects false premises less often than the base model, 5 times in 10 against 7 in 10. Shorter answers did not improve every behaviour. The technical report gives the full method.

What you can do with any chatbot

The same caution applies to any chatbot, large or small. Ask for short answers, which leaves fewer claims to check. Ask the model to separate what it knows from what it is guessing, and treat a confident reply with no such separation as a draft. Then check any name, number or date you would act on against a source you can open. Our guide to checking an AI answer without starting over walks through that step, and clear prompts in English and Turkish shows how to ask for a short, specific reply in either language.

No single agreed definition

There is no single agreed definition of AGI. Four families of definition come up again and again, and each one counts a different kind of evidence. Looking at what each would count as proof, and where each one breaks, is more useful than memorising a slogan. The four are imitation, useful work, graded levels and skill-learning efficiency.

The choice matters in practice. A system that performs well on a school-style exam may still fail a puzzle it has never seen, and a system that sounds convincing in conversation may not complete useful work reliably. Each definition rewards a different strength, and each one leaves a different weakness hidden.

Imitation and useful work

The oldest route comes from Alan Turing. In “Computing Machinery and Intelligence” (1950), published in the journal Mind, he replaced the question “Can machines think?” with an imitation game: a judge exchanges typed messages with a person and a machine, and tries to tell which is which. The evidence is conversational. If the judge cannot reliably tell the two apart, the machine has passed for a person in that exchange. The weakness is that imitation can be gamed. A system can be tuned to sound human without the understanding the test was meant to reveal, and a judge can be swayed by style alone.

The second route asks what a system should do for people. The OpenAI Charter, the company’s public statement of its mission, says that by AGI it means “highly autonomous systems that outperform humans at most economically valuable work”. This definition is practical, because you can ask whether a system does a job as well as a person does. Its weakness is that economically valuable work changes as jobs appear and disappear, and the wording says nothing about how the work is done. A system could meet it through a narrow set of tasks and still be fragile outside them.

Levels and skill-learning efficiency

Morris and colleagues at Google DeepMind, in “Levels of AGI” (2023), grade systems on two separate axes. Performance runs from “Emerging” up to “Superhuman”. Generality runs from narrow to general. The paper also treats autonomy as a separate question rather than folding it into the levels. The evidence is a graded scale that can be checked against agreed tasks. Its weakness is the phrase “agreed tasks”: a level is only as meaningful as the task list behind it, and someone has to choose that list and keep it current.

François Chollet’s “On the Measure of Intelligence” (2019) takes a different angle. It argues that intelligence is better measured by how efficiently a system acquires new skills from little experience. It also introduces ARC, a benchmark of novel grid puzzles that a system is not meant to have seen before. The evidence is performance on tasks designed to be unfamiliar. The weakness is that ARC measures one kind of novelty, built from simple coloured grids. Doing well on it does not, by itself, show competence in language, science or everyday planning.

Why benchmarks age

Any test loses value once it becomes a target. When its questions are posted online, they can end up in later training data, and scores can rise without the underlying ability rising. Researchers call this contamination. You can only check for it if you know what a model was trained on, and many models do not publish that in full.

Flyingkiwi’s frozen test sets were never uploaded or trained on, which rules out one route to contamination. It does not rule out all of them. The base model, Qwen3-0.6B, was pretrained on data that is not public, so even this project cannot say whether test-like text appeared there. The open data article explains why that gap matters.

Where a small model sits

Flyingkiwi’s English model scored 63 out of 100 on its frozen set of project-written cases, the same as the unchanged Qwen3-0.6B it was built from. The grading was done by AI graders against reference answers, so treat that number as an estimate rather than a human verdict. On TruthfulQA MC1, a set of 817 questions where common misconceptions are the tempting wrong answers, it scored 25.1%, against 25.8% for the base model. The gap is inside the test’s sampling error of about 1.5 points, so the honest reading is that the two are roughly level.

The model cannot browse the web, use tools, see your files, remember earlier conversations or know today’s date. Against any of the definitions above, it is nowhere near AGI. It is a narrow tool, useful for some tasks and unreliable for others, and it is best judged as one. The article on what small models can do covers its strengths and failures in more detail.

What is still missing

The four definitions point to gaps rather than dates, so it is more useful to name the gaps than to predict when they will close. The first is reliable facts. A model trained on text can state a wrong thing as fluently as a right one, which is the basic problem behind invented details. A reader meets this whenever a fluent answer is wrong about a name, a date or a figure. The second is knowing what you do not know. Flyingkiwi’s English model said it could not know the answer in 18 of 20 cases where that was the right response. Yet 6 of its 100 replies still contained invented facts or claims.

The third gap is learning a genuinely new task from a few examples, which is the kind of skill Chollet’s measure cares about. This project has no measurement of it, so no number appears here. The fourth is long, multi-step plans, where an early mistake carries into later steps. The project has not measured plans of that length either. The fifth is memory across time. The model does not remember earlier conversations at all, so every new chat starts from nothing.

Why the debate matters to you

Definitions are not only an academic matter. They can shape what regulators ask for, where investment goes and which products get built. A claim that a system is “general” usually depends on the definition behind it, which is one reason to ask what was measured and how. The numbers in this article each measure something narrower than AGI. The 63 out of 100 result scores answers to 100 project-written cases, and the TruthfulQA result scores multiple-choice answers to questions that people often get wrong. Neither is a general intelligence score, and neither should be quoted as one.

You do not need to settle the definition to judge a claim. Ask which test was used, who graded it, and whether its questions were public before the system was tested. The same habits that help you check a chatbot’s answer, covered in how to check an AI answer without starting over, apply to an AGI announcement too.

What makes a dataset open in practice

Four tests help. Can you obtain the data without a private arrangement? Does its licence allow the use you have in mind, and what does it ask in return? Is it documented, so that someone has said what it contains, who made it and what it leaves out? Can you pin the exact version you used? Gebru and colleagues’ “Datasheets for Datasets” (2018) proposes that every dataset come with a short datasheet covering its motivation, composition, collection process and recommended uses.

The last test is easy to skip. A file can be downloadable, but if nobody recorded which revision was used, a result built on it cannot be rebuilt later, even by its own authors.

What Flyingkiwi was trained on

Flyingkiwi’s English and Turkish models were both fine-tuned from Qwen3-0.6B. The table lists every source used, with its licence and the job it did. All of the sources are public, their revisions are pinned, and no private conversations or user data were used.

Three rows need a sentence each. The Aya dataset supplied prompts only, not its answers. The teacher’s answers are not a dataset in the usual sense: an open-weight model wrote them under a written set of rules, and those rules are part of the documentation. The frequency counts from the web and code corpora were used only to choose which tokens to keep in the vocabulary, not to teach the model text.

Reproducibility and auditing

A pinned revision lets someone rebuild the same prompt set. The English fine-tuning used 15,834 training and 344 validation conversations. The Turkish fine-tuning used 20,778 training and 388 validation conversations. Those counts mean little unless the sources are pinned, because a result you cannot rebuild is, in the end, only a claim. Processing needs recording as well as sources. In the concise rewrite, the teacher rewrote 22,067 English answers, with 43% of the characters kept, and 30,036 Turkish answers, with 33% kept. A rewrite that dropped a code block, added a disclaimer, grew longer or switched language kept the original answer. A list of sources alone would not tell a reader that.

Known training data also makes an audit possible. You can search the training set for test questions and see whether any leaked in. Flyingkiwi’s frozen test sets were never uploaded or trained on. The base model’s pretraining data is not public, though, so exposure there is unknown. That is a real limit: the audit covers the project’s own data, not the foundation beneath it. The guide to evaluating a chatbot honestly describes the other habits that keep test sets clean.

Licences need checking one source at a time

Apache-2.0 and MIT both allow reuse on conditions, such as keeping the licence notice. Share-alike licences work differently. The earlier research corpus, labelled v2, used databricks-dolly-15k, which is licensed under CC BY-SA 3.0. That material is not part of v3 or v3.1. Share-alike material is not automatically relicensed under Apache-2.0, so using it would need a separate decision, and that is one reason it was left out. The licences in the table are not all the same. Most rows are Apache-2.0, the GSM8K row is MIT, and the frequency-count sources are ODC-BY or Apache-2.0. Each one carries its own conditions, so each needs its own check rather than a single answer for the whole project.

The practical rule is simple to state and tedious to apply: read the licence of each source, not just the licence of the project that uses it.

Languages depend on the data that exists

Open multilingual data decides which languages a small model can serve. Turkish has less public conversation data than English. The Turkish model therefore drew on prompts from Aya, a community-built multilingual dataset of human-written prompts and answers, and on OASST2 prompts translated into Turkish. Flyingkiwi used only the Aya prompts. The Turkish training set lists 7,761 human-written prompts from OASST2, Aya and translated OASST2, against 7,455 teacher-written requests, so the teacher-written share is close to the human-written one.

Even with that data, the Turkish model scored 37 out of 100 on its frozen set, against 12 for the unchanged base model. That is a large gain and still a weak result, as the article on why Turkish is hard for small language models explains. Open data is necessary for a language like this, but it is not sufficient. The data still has to be cleaned and used with care. Grading used AI graders with spot checks, and there was no independent native-speaker panel, so the results still need checking by people who read Turkish natively.

Open weights are not an open pipeline

Flyingkiwi publishes its open weights under Apache-2.0 and documents its fine-tuning data, so anyone can run the models and inspect how they were trained. The base model’s pretraining data is not public, though. That makes Flyingkiwi’s release open weights, which is a different thing from a fully open training pipeline.

The Open Source AI Definition 1.0, published by the Open Source Initiative in 2024, asks for detailed information about the training data, enough for a skilled person to build a substantially equivalent system. Flyingkiwi documents its own fine-tuning data, but it cannot describe pretraining data it did not collect. On that point the gap is inherited from the base model, and this project cannot close it.

Why the stakes are larger than one small model

A small model makes the problem easy to see. Larger systems face the same question with higher stakes. As systems become more general, what they learn is shaped by the data they see, and by who is able to inspect that data. If the data is closed, the people a system affects often cannot check what it was taught, or test whether it works for their language, their subject or their community. Even a well-documented dataset reflects the choices of the people who collected it, and documentation is what lets others see those choices.

Open, documented data does not guarantee a fair or accurate system. It does make claims checkable, which is the precondition for either. As a reader, the habit is the same as for any AI answer: ask what the system learned from, and whether you could find out for yourself.

The trade: download size against answer quality

Making the weights smaller is the most direct way to make a model faster to download. The cost is that every weight becomes a rougher approximation of the value training gave it. Storing weights with fewer bits is called quantisation, and the useful question is always how much answer quality each saving costs.

For English, the options were 8-bit weights at 561 MB, plain 4-bit weights at 341 MB, and several mixed versions in between. In English, only the 8-bit version matched the unquantised checkpoint to the three decimal places we report. The rest of this article shows what the smaller files lost, and why one number from one test is not enough to choose between them.

What quantisation does, concretely

A 16-bit float can represent a very wide range of values. A quantised weight is stored as a small integer, and a scale turns that integer back into an approximate value. Flyingkiwi’s browser weights store 8-bit integers with one 16-bit float scale for every 32 values, so each scale is shared by a small group of neighbouring weights rather than by a whole matrix.

The difference between 8 and 4 bits is the number of steps available. An 8-bit integer has 256 levels, and a 4-bit integer has only 16. Imagine two rulers covering the same length, one marked in 256 steps and one in 16. Whatever you measure is rounded to the nearest mark, so the coarser ruler leaves larger errors. A model multiplies hundreds of millions of these rounded values, and small errors in many weights can add up to a different answer.

How the models were measured

The main yardstick is held-out answer loss, measured in nats per token. A held-out conversation is one the model did not train on. The loss tells you how well the model predicts the words of the assistant’s reply, so a lower number is better. The table below uses 40 English and 24 Turkish validation conversations.

Why only the assistant’s answers? A chat model is used to write the assistant’s turns, so the loss on those turns is closer to what a reader experiences. A general perplexity score averages over all kinds of text, including text the model is never asked to write. That average can hide a change in exactly the part of the conversation that matters.

The validation set is small. Treat the decimal places as a guide to direction, and rely on the blind tests further down to show whether a change reaches the answers.

Reading the table

In English, 8-bit storage cost 0.0003 nats per token. The float checkpoint and the shipped file both score 1.327, so at this precision the rounding made no visible difference. Turkish moved from 1.562 to 1.565 in the same comparison. The full set of measurements is in the technical report.

Plain 4-bit rounding cost 0.17 nats per token in English, and the Turkish loss rose from 1.565 to 1.804. GPTQ, the method described by Frantar et al. (ICLR 2023), roughly halved that cost in English, bringing the 4-bit English loss to 1.407. It quantises one layer at a time, column by column. After each column is rounded, the method pushes that rounding error onto the columns not yet quantised, using statistics from the layer’s inputs to decide where the error goes.

Keeping the most sensitive matrices in 8 bits brought the 420 MB English model to a loss of 1.365. It is still more than a hundred times the cost of 8-bit weights, at 0.039 nats per token. Sensitivity was ranked with the diagonal Fisher information of the answer loss, and the matrices kept in 8 bits were mostly the small value and key projections.

Does loss predict answers?

Loss is a proxy. To check whether a change matters, we ran blind tests on 100 project-written cases per language. Answers from two models were mixed, and an AI grader compared each answer with a reference answer without knowing which model had written it. The grader counted correct answers and, separately, replies that contained invented details.

The English models with 4-bit weights, including the mixed version, lost between 4 and 9 points. In each of those rows the number of invented-detail flags went up. The shipped 8-bit model scored the same as the unchanged reference, with the same number of flags.

The reference total moves by a few points between runs, from 62 to 66 in English, so differences of two or three points are within grader noise. Each category has 10 to 30 cases, and one case moves a category score by 3 to 10 points. The shipped variant was chosen with this same test, so its score is slightly optimistic.

The QAT trap

The most useful lesson came from the Turkish v3 model, and it nearly led us astray. On that model, plain 4-bit rounding looked almost free: 1.5425 nats per token against 1.5401 at 8 bits. GPTQ looked harmful, at 1.5965. Someone who saw only those numbers might conclude that plain rounding is the better method.

The explanation is how that checkpoint was trained. The v3 Turkish model had been trained with 4-bit quantisation-aware training on exactly that rounding grid. Plain rounding therefore reproduced the model that training had produced, while GPTQ moved the weights away from that grid. The comparison was measuring how well each method matched a grid the model had already learned, not which method preserves an ordinary model best.

Without quantisation-aware training, as in v3.1, the picture reverses. Plain 4-bit rounding raised the Turkish loss from 1.565 to 1.804, and GPTQ raised it to 1.664. The rule is simple: check how a checkpoint was trained before reading quantisation results. A result on a model trained with QAT says something about that training, and little about rounding an ordinary checkpoint. How the v3.1 models were trained instead is explained in the knowledge distillation article.

What would change the decision

v3.1 ships 8-bit weights, so the English browser download is 571 MB including the engine, tokenizer and prompt cache. The 561 MB of weights is the largest part of that download. The engine that reads these files is described in the WebAssembly article, and it already supports 4-bit weights.

The open route is quantisation-aware training of v3.1 on its own 4-bit grid. That has not been tested, so the 341 MB figure is not a result for this model. The open question is the trained model, not the code. Until a version of that kind passes blind tests against the unchanged reference, 8 bits remain the release.

A checklist for quantising a small model

If you are quantising a small model yourself, these steps address each problem described above.

1. Find out how the checkpoint was trained. A model trained with QAT can make plain rounding look better than it is for an ordinary checkpoint.
2. Measure on the assistant’s answers, not only on general text.
3. Measure the format you will actually ship, with its real scales and group size.
4. Compare against the unchanged model at the same settings.
5. Run blind tests before release, and count invented details separately from correct answers.

The blind-test method is covered in more detail in how to evaluate a chatbot honestly. The conclusions here belong to one 0.6B model and one set of cases, so treat them as a worked example rather than a rule for every model.

What one token costs

Every weight is read once for each token generated, and every read feeds a multiplication. A 0.6B model is therefore limited less by clever arithmetic than by how much data must move and how many multiplications can be done per step. A smaller weight format means less to read, and a vector instruction means several multiplications in a single step.

The site uses two inference worker scripts, so the calculation runs off the main page thread. The answers never leave the browser. Everything described below is what happens inside the tab after the model has been downloaded.

A small engine with no dependencies

The engine is about a thousand lines of C with no dependencies. It is compiled with clang straight to WebAssembly, without Emscripten, and the same source builds natively for evaluation on a laptop. That second point matters. Tests run the same arithmetic that the browser runs, so a measured result describes the code that ships rather than a separate reference implementation.

The engine runs on the CPU. It needs no GPU and no browser extension, but it does need a current browser with WebAssembly SIMD, such as recent Chrome, Edge, Firefox or Safari. Only the two inference worker scripts are allowed to compile WebAssembly, and the site’s Content Security Policy enforces that. Other scripts on the page cannot compile WebAssembly, which keeps the compiled code in one known place.

Integer kernels for 8-bit weights

The weights are 8-bit integers with one scale for every 32 values. The activations, the intermediate numbers that flow between layers, are also quantised, in groups of 32 values. That lets the core matrix products use integer arithmetic, which processors handle quickly, instead of floating-point arithmetic. The products use SIMD integer dot products from WebAssembly SIMD128.

Where the browser supports relaxed SIMD, one instruction, i32x4.relaxed_dot_i8x16_i7x16_add, performs 16 multiply-adds at once, with 7-bit activations. The relaxed SIMD proposal defines the instruction. Where that instruction is not available, the engine uses the ordinary SIMD128 path.

Prefill is the step that reads your whole prompt before the first reply token appears. During prefill, the engine unpacks each weight group once and reuses it for four tokens. Generation produces one token at a time, so it cannot share weights across tokens in the same way.

Threads and cross-origin isolation

The engine shares memory between up to six worker threads, and that needs SharedArrayBuffer. The MDN security requirements describe the condition: the browser makes SharedArrayBuffer available only to cross-origin isolated pages. The site therefore sends Cross-Origin-Opener-Policy as same-origin and Cross-Origin-Embedder-Policy as require-corp.

Work is handed out through an atomic counter. Matrix rows and attention heads go to whichever thread is free next, which suits a phone with a mix of fast and slow cores. Threaded results are bit-for-bit identical to single-threaded ones, so the thread count changes the speed of a reply but not its content.

The practical consequence is strict. Under cross-origin isolation, every resource the page loads must be same-origin or must opt in explicitly, for example with a suitable Cross-Origin-Resource-Policy header. The MDN page on Cross-Origin-Embedder-Policy sets out the rule. Anyone building a similar page should plan for it from the start, because a third-party resource without that opt-in is blocked. It is also why every file on this site, fonts included, is served from the site itself.

Memory: a compact cache and a fixed start

The attention cache stores keys and values as 8-bit numbers, with one scale for every 32 values. This keeps Qwen3’s outlier key channels accurate. The first four positions, known as attention sinks, stay in full precision, because the model relies on them heavily.

The fixed system prompt is cached ahead of time and shipped with the model as system.kv. Each new turn only processes what has changed since the previous one, so the first reply does not wait for the instructions to be read again. Long conversations still get slower, because there is more context for each new token to attend to.

The context window is 4,096 tokens. On devices that report 4 GB of memory or less, it is 2,048 tokens, which halves the cache. In the measured runs, peak tab memory ranged from 1,113 MB to 1,358 MB. The live public-site run peaked at 1,286 MB.

Getting the weights into the browser

The first message downloads the model for the chosen language. The full English download is 571 MB (571,378,688 bytes), including weights, engines, tokenizer and prompt cache. The Turkish download is 567 MB (566,790,616 bytes). The local AI article explains what that download means for a visitor.

The download is split into 17 parts for each language. Each part is checked by size and SHA-256 hash before it is used, and verified parts are kept in the browser’s Cache Storage. A later visit normally reuses them, unless the browser has cleared its storage.

Speed on a laptop and on emulated phones

The measurements came from headless Chrome 155 on an AMD Ryzen 5 7530U laptop. The phone rows are emulation. The processor was limited to about half or a quarter of a laptop core, and the memory was capped. They are not measurements on real phones, and real devices will differ.

On the public site, a real English reply arrived after 71.5 seconds, including the 571 MB download. It ran at 25.7 tokens per second with relaxed SIMD and six threads, and peak tab memory was 1,286 MB. That is one run on one device and one connection, so read it as an example rather than a benchmark. Speed depends heavily on the device, and the emulated low-end rows show how much a weaker processor changes the wait.

Is the browser build faithful?

A simple check of the browser build uses KL divergence, which measures how far one probability distribution is from another. Here it compares the engine’s distribution over the next token with the one from the checkpoint. A value of zero would mean the two are identical.

On 30 development prompts, measured at the first reply token, the released English file stays within a mean KL divergence of 0.0114 of its checkpoint. The WebAssembly relaxed-SIMD build measures 0.0147. Both values are small, but they are not zero, so the browser build is close to the checkpoint rather than identical to it. The Turkish figures are higher: 0.0513 for the released file and 0.0563 for the relaxed-SIMD build, so Turkish replies carry a little more drift.

Fidelity is only one check. A faithful engine can still run a model that gives weak answers, so the quality results in the technical report remain the measure of quality. The limits are plain: a 0.6B model knows little, long conversations slow down, the download is about 570 MB per language, and speed varies widely between devices.

The idea: a teacher’s output trains a student

Distillation starts from a simple observation. A teacher model that is large enough to answer carefully is expensive to run for every question. A student model that is small enough to run on a device is cheap, but it has less capacity and so it knows less. Distillation trains the student to behave like the teacher, using the teacher’s outputs as the training signal.

Depending on the method, the student learns from the teacher’s output probabilities or from the text the teacher wrote. The two approaches are different in what the student receives, and the difference matters for how you design the training data.

Two flavours: soft targets and written answers

The classic version, described by Hinton, Vinyals and Dean in 2015, trains the student to match the teacher’s full probability distribution over possible next tokens. Those probabilities are called soft targets. They carry information about which wrong answers were nearly right, which a single correct word does not.

Sequence-level distillation, as described by Kim and Rush in 2016, is simpler. The student is trained on sequences that the teacher produced. Flyingkiwi used this second approach. Its students learned from the teacher’s written answers, not from the teacher’s probability distributions, which is the practical reason the rules for those answers matter so much.

The teacher and its rules

The teacher was Qwen3.5-35B-A3B, an Apache-2.0 model run in FP8. It wrote answers under a written rule set. The answers had to be accurate, say when the model was unsure or when something was unknowable, correct false premises, ask when a request was ambiguous, and use Markdown and code blocks.

The rules matter because a student copies style as well as facts. If a teacher hedges without reason, invents detail or pads its answers, the student learns those habits too. The students then trained on those answers with the short system prompt that the app sends, so the training matched the way the app is actually used.

What the teacher was asked

The English set had 15,834 training and 344 validation conversations. They were drawn from several kinds of prompt:

• 3,369 human-written prompts from OASST2 and Aya;
• 653 GSM8K word problems;
• 6,292 everyday, explanation, writing, coding and maths requests written by the teacher;
• 1,890 honesty probes;
• 479 questions about invented entities;
• 125 identity questions;
• 248 memory checks;
• 1,198 chained conversations.

Of the English conversations, 1,544 continue with teacher-written follow-up turns, and 2,385 have no system prompt at all. The Turkish set had 20,778 training and 388 validation conversations. It included 7,761 human-written prompts from OASST2, Aya and translated OASST2, 433 GSM8K word problems, 7,455 teacher-written requests, 1,941 honesty probes, 460 invented-entity questions, 118 identity questions, 474 memory checks and 1,567 chained conversations.

The honesty probes teach the student to admit uncertainty when a question has no reliable answer. Invented-entity questions ask about things that do not exist, so the honest reply is to say that the thing is unknown rather than to describe it. Memory checks and chained conversations train the student to use earlier turns in the same chat. Each category targets a behaviour that a small model is likely to get wrong if nobody trains it.

Calibration from the student’s own knowledge

Some training examples teach the model to say when it does not know. These were built from TriviaQA questions. The base model, Qwen3-0.6B, was asked each question, and the questions were sorted by whether it could answer them. Known answers were kept, and unknown ones were taught with an honest “I’m not sure”.

The reason to build calibration from the student’s own knowledge is that the uncertainty to teach depends on what the student actually knows. A teacher with more knowledge would mark many questions as answerable, even though the small model cannot answer them. The student would then learn to answer with confidence where it has no support.

Quality control: filtering and the concise rewrite

GSM8K gold answers were used to filter the teacher’s answers to maths problems. Cleaning for v3 filtered or reordered answers that contained unprompted remarks about the assistant’s own limits, thinking aloud, or maths results stated before the working.

The concise rewrite was new in v3.1. The v3 answers were two to four times as long as the base model’s, and blind grading flagged more invented details. Asking for brevity in the system prompt did not shorten them. So the teacher saw each whole conversation and rewrote every assistant message into the shortest complete answer.

The rewrite had to keep code blocks, requested formats, honest statements of uncertainty, and anything a later turn refers to. A rewrite was rejected if it dropped a code block, added a disclaimer, grew longer, or switched language. In English, 22,067 answers were rewritten and 43% of the characters were kept. In Turkish, 30,036 were rewritten and 33% were kept. The reasoning is that every extra sentence is another chance for a 0.6B model to state something false, as the article on invented details describes.

The recipe and the compute

Training was full-parameter fine-tuning in BF16 with FP32 master weights, for two epochs. The peak learning rate was 1.5e-05 with cosine decay. No quantisation-aware training was used for these models.

English took 148 steps, and its validation loss fell from 2.165 to 1.311. Turkish took 300 steps, and its validation loss fell from 2.610 to 1.632. These are the training run’s own validation losses, so they are not the same measurement as the answer-loss table in the quantisation article.

The compute was one NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB, in an existing Colab Pro+ session. Rewriting took 57 minutes. Training took 7 minutes for English and 15 minutes for Turkish. The whole Colab session lasted about two hours.

What the results show, and what they do not

The student is limited by what it can hold. A 0.6B model cannot store everything a 35B teacher knows, and it copies the teacher’s mistakes along with the good answers. Distillation does not add a teacher’s knowledge for free.

On the frozen test sets, English scored 63 out of 100, the same as the unchanged Qwen3-0.6B. Turkish scored 37 out of 100, against 12 for the base model. English also corrected false premises less often, at 5 out of 10 against 7 for the base model. Turkish quality was judged by an AI grader rather than by native speakers, and 37 out of 100 is low in absolute terms. The article on what a 0.6B model can do covers the failures in more detail.

The reasonable reading is that distillation mostly changed behaviour, such as honesty, brevity and language, rather than adding knowledge. English v3.1 said it could not know something 18 times out of 20, against 17 for the base model. Turkish v3.1 said so 14 times out of 20, against 1. On answerable facts and reasoning, English v3.1 scored 20 out of 30 against the base model’s 22, so no knowledge was visibly added. This is an interpretation of the numbers, not a proven fact, because the experiment did not isolate which training data caused each change. The technical report lists the training details.

Why most comparisons mislead

The usual problems are easy to name. Examples are picked because they flatter one side. The test set changes from one run to the next, so nobody can repeat the result. The decoding settings, which control how the model picks each word, are not reported. The grader knows which model wrote which answer. And scores are given as percentages with no denominator, so a score from a handful of questions looks the same as one from a large set.

1. Freeze the test before training

Flyingkiwi’s test is a frozen test set of 100 project-written cases per language. The cases were never used for training, and the categories are fixed: 30 answerable questions, 20 the model cannot know because they concern live, private or future information, 10 false premises, 10 ambiguous requests, 10 friendly messages and 20 multi-turn follow-ups. Each category tests a different thing. Unknowable questions reward honesty, because the right answer is that the model does not know. Ambiguous requests reward a clarifying question. Multi-turn follow-ups check whether the model keeps track of the chat.

Keep the set frozen. Once you have studied its failures and changed the model to fix them, the set is no longer independent of your work. Flyingkiwi’s frozen sets were never uploaded or trained on, but the project still notes that choosing a release with the same test makes its score slightly optimistic.

2. Compare against an unchanged reference

A comparison needs a reference that you did not change. Flyingkiwi compares each release with unchanged Qwen3-0.6B, run under the same conditions: the same system prompt, greedy decoding and a repetition penalty of 1.1. The reference is the 8-bit file, which stays within a mean KL divergence of 0.0014 of the full-precision model. KL divergence is a measure of how far one model’s predictions drift from another’s, so a low value means the reference is a faithful copy and not a damaged one.

Then run the exact file you ship. A test on a full-precision model tells you about a different product. Flyingkiwi’s comparisons used the released 8-bit file in the native build of its engine. The native build uses the same source as the browser engine, but it is not the browser build itself, and that difference should be stated when you report results.

3. Grade blind

Mix the answers from both models at random, so the grader cannot tell which reply came from which model. This is blind evaluation. Flyingkiwi’s grading used AI graders that compared each answer with a reference answer, with spot checks. Count invented-detail flags separately from correctness. A reply can be mostly right and still contain a made-up name or number, and that is a different failure from a wrong answer.

Blind grading is only as good as the grader. Flyingkiwi’s graders were AI systems, and there was no independent human or native-speaker panel. Treat a Turkish score in particular as a rough signal rather than a verdict on fluency. The same caution applies to any test you run yourself, however carefully you hide the labels.

4. Know your noise

Run the same test twice and you will see different numbers. Flyingkiwi’s English reference moved between 62 and 66 across runs, so a difference of two or three points is within the noise. Categories have 10 to 30 cases, and each case moves a category score by 3–10 points. A single case changing its verdict can look like a trend.

Choosing a release with the same test makes its score slightly optimistic, because part of the choice was made by looking at that score. A public benchmark helps as a second check. The TruthfulQA paper by Lin, Hilton and Evans describes questions on which some people would answer falsely because of a common misconception. Its MC1 test has 817 questions, and its sampling error is about 1.5 points. It should supplement your own cases, not replace them, because it measures something narrower than your use.

5. Check your settings, not your intentions

Decoding settings change answers, and it is easy to believe you set them when you did not. In Hugging Face Transformers, greedy decoding set only in a generation config can be overridden by the model’s own sampling defaults. The safe pattern is to pass use_model_defaults=False, do_sample=False and num_beams=1 directly to the generate call, then check the effective values.

Flyingkiwi’s 24 October 2026 protocol correction is an example of this problem: the reference model’s answers turned out to have been sampled while the candidate’s were greedy, so the comparison was uneven. The historical counts were kept and labelled as preliminary observations from mismatched settings rather than hidden. The 24 October correction explains what changed.

6. Report the losers

A report that shows only winners tells you little. Flyingkiwi’s reports include rejected variants. An English fine-tune trained on unfiltered data scored 55 where the reference scored 65. A blend that kept only 25% of the fine-tune scored 53 against 64. Both were rejected. The Turkish model’s result is stated plainly as well: 37 of 100, against 12 for the base model, which is a clear gain and still a weak score.

The same discipline applies to size trade-offs. Blind tests of 4-bit models lost 4–9 points, so the v3.1 release ships 8-bit weights. The 8-bit and 4-bit weights article covers that trade-off in detail. Report the losers in your own tests too. If you publish only the variant that won, readers cannot tell whether the win was a result or a selection.

Limits of this method, and a checklist

This method has limits, and you should state them next to every number. The graders are AI systems. There is no native-speaker panel. Each set has 100 cases, and the release choice was made with the same test. None of this makes a comparison worthless. It means a score is a measurement with edges, not a verdict.

Before you publish a comparison of your own, check that you have:

1. a test set kept out of training, with any reuse for choosing stated;
2. categories that test different things, including unknowable questions, false premises and multi-turn context;
3. an unchanged reference run under the same system prompt and decoding settings;
4. the exact file you plan to ship, with its effective settings checked;
5. blind grading, with invented details counted separately from correctness;
6. a repeat run, so that small differences are not read as results;
7. every variant you tried, including the ones that lost, with its counts out of the total.

Words built from suffixes

Turkish is agglutinative: a word is built by adding suffixes, one after another, to a root, and each suffix adds one grammatical meaning. Take evlerinizden, which means “from your houses”. The root is ev, house. The suffix -ler makes it plural, -iniz makes it “your” in the plural or polite form, and -den means “from”. A phrase that takes three English words is one Turkish word.

A longer example is gidemeyeceklermiş. Its root is git, go, which becomes gid- before a vowel. The -eme- part is the usual negative ability marker, so it means cannot. A linking consonant, -y-, follows, then -ecek marks the future, -ler means they, and -miş marks something reported or only apparently true. The whole word means something like “apparently they will not be able to go”.

The tokenizer does not follow these boundaries. It split evlerinizden into ev, ler, iniz and den, which matches the grammar. It split gidemeyeceklermiş into gid, em, eye, cek, ler and miş, which cuts across it. The model works with these pieces, so a single grammatical unit can be spread over several of them.

Vowel harmony keeps suffixes in step

Turkish vowels fall into two groups. Front vowels are e, i, ö and ü. Back vowels are a, ı, o and u. Most suffixes take a form that matches the vowels of the root, a pattern called vowel harmony. The plural of ev is evler, with front vowels, while the plural of kapı (door) is kapılar, with back vowels. The “from” form is evden for ev and kapıdan for kapı.

The rule is regular, but a model must learn every combination. The plural and the “from” form of just two nouns already give four different surface forms. Multiply that by every noun, verb and tense, and the learning task is large for a model of about half a billion parameters.

More tokens for the same meaning

A model reads text as tokens, which are pieces of words, and it writes one token at a time. Measure the same meaning in both languages and the Turkish version is longer. The sentence “Yarın sabah saat dokuzda toplantıya katılamayacağımı öğretmenime bildirmem gerekiyor.” has nine words and takes 27 tokens with the Turkish tokenizer. Its English translation, “I need to tell my teacher that I will not be able to attend the meeting at nine tomorrow morning.”, has twenty words and takes 21 tokens with the English tokenizer. With the English model’s trimmed vocabulary, the Turkish sentence takes 37.

That works out to about 1.3 times as many tokens for the same meaning, roughly three tokens per Turkish word against about one per English word. This is one example, not a measure of all Turkish text. The consequences are practical: a reply needs more steps to generate, the context window fills faster, and the same content takes longer to appear on screen.

Where the training data came from

The Turkish model combined several sources. Its training set has 20,778 conversations, with 388 kept for validation. The human-written prompts came from OASST2, Aya and translated OASST2 prompts, 7,761 in all. Another 7,455 requests were written by the teacher model, and 433 were GSM8K word problems.

The rest of the mix targets behaviour the base model lacks. It includes 1,941 honesty probes, which teach the model to say when it does not know, and 460 questions about invented entities, which test whether it will make things up. There are also 474 memory checks and 1,567 chained conversations, which train it to keep track of earlier turns.

What the Turkish model gained

The base model mostly fails in Turkish. On the 100-case frozen set, Qwen3-0.6B answered 12 cases correctly, and the Flyingkiwi Turkish model answered 37. The table shows where the gains came from, with the number of cases in each category.

The largest change is in honesty about the unknown. On the 20 questions the model cannot answer, the base model said so once, and Flyingkiwi 14 times. The gains are real, but 37 out of 100 is low in absolute terms, and the grading was done by an AI system rather than a native speaker.

Where it still fails

The Turkish model still gets arithmetic and facts wrong often. Its replies with invented facts or claims numbered 17 on the frozen set, against 19 for the base model, so the improvement there is small. Its answers are also longer than the base model’s, averaging 92 tokens against 74, even after the rewrite.

Engine fidelity is lower than in English too. Fidelity here means how closely the browser engine’s output probabilities follow the model’s checkpoint, measured as mean KL divergence at the first reply token across 30 development prompts, where lower is closer. The Turkish figure is 0.0513, against 0.0114 for English. It is a weaker match, and it belongs next to any Turkish score.

A vocabulary built for Turkish

The Turkish model keeps 78,255 tokens from the base model’s 151,669. Rare entries were removed and the byte-level BPE merges were rebuilt, but text made only of kept tokens is still tokenised exactly as before. The 256 byte symbols remain, so nothing becomes unencodable. On held-out Turkish text, the trimmed vocabulary needs 0.21% extra tokens. The Turkish download is also slightly smaller: 567 MB against 571 MB for English.

Practical tips for Turkish users

Short, concrete questions work best with this model. Ask for one thing at a time, and check every number before you use it. Give the names, numbers and dates you already know, so the model does not have to guess them. The guide to checking AI answers shows how to verify a claim. If you need code, or a technical term is clearer in English, switch languages. Both models are on the site, and the clear prompts article has examples in both languages.

For the wider picture, see what a 0.6B language model can and cannot do.

What 0.6B means

“0.6B” means about 600 million parameters. A parameter is one of the learned numbers inside the network, and the count is a rough guide to how much a model can store. Flyingkiwi’s English model has 528 million parameters after its vocabulary was trimmed, and the Turkish model has 521 million. Large general assistants are far larger than this. The gap shows most in breadth, and in fine detail such as exact figures and obscure names.

In the browser, the weights are stored as 8-bit integers. The English weights take 561 MB. The full download, including the engine, the tokenizer and a cache of the fixed system prompt, is 571 MB for English and 567 MB for Turkish. Either language is therefore a download of about 570 MB, which is worth knowing before the first message.

Why choose a small model anyway

The case for a small model is practical, not a claim that it is as capable as a large one. It runs on your own device, and your messages are never sent to a server. It needs no account, it works after one download, and it has no cost per message. The article on local AI in your browser explains what that download involves and what stays on your device.

What it does reasonably well

On Flyingkiwi’s English frozen set, the model got 20 of 30 answerable questions right. It said it could not know the answer in 18 of 20 cases where that was the right response, kept a friendly conversation going in 6 of 10, and kept track of context in 11 of 20 multi-turn follow-ups. It handled ambiguous requests in 3 of 10 cases, where the unchanged base model managed none.

These strengths are real but narrow. Each category has only 10 to 30 cases, so a single case moves a category score by 3 to 10 points. The categories test different things. The answerable questions have known answers. The “cannot know” questions ask about live, private or future information, where the honest reply is to say so. The false premise questions contain an assumption that is wrong. The grading was done by AI graders against reference answers, not by a panel of people, and the results describe one release. Treat them as a useful signal, not a verdict.

Where it fails

Small models fail in predictable places. Arithmetic is unreliable, and so are niche facts, precise numbers and anything recent. The model does not know today’s date, cannot browse, cannot use tools, cannot see files you attach and cannot remember earlier conversations. It can produce a hallucination, a fluent claim with no support behind it, as confidently as a correct answer, which is why its answers need checking. The riskiest case is a confident, precise figure, such as a date or a measurement, that looks checked but is not. Precise numbers deserve the same scepticism as names.

Two results are sobering. The English model corrected false premises in 5 of 10 cases, down from 7 of 10 for the base model. On TruthfulQA MC1, a multiple-choice test of 817 questions, it scored 25.1%, against 25.8% for the base model, and that gap is within the test’s sampling error. Turkish is better than the base model, which managed 12 out of 100, but 37 out of 100 is still weak, with frequent errors in facts and arithmetic.

Good uses and bad uses

The model suits tasks where you can see the result and judge it quickly. Each example below produces output you can read and correct in seconds, which matters for a model that makes regular mistakes. Some good uses are:

• Rewording a paragraph you wrote, for example “Make this paragraph shorter, and keep every date.”
• Drafting a short message, for example “Write a two-line reply saying I can join on Thursday but not on Wednesday.”
• Explaining a common concept, for example “Explain what a hash table is in three sentences.”
• Writing a small code snippet, such as removing duplicates from a Python list. This is one of the app’s own example prompts.
• Practising a simple conversation in English, for example ordering a coffee.

Bad uses matter just as much. Do not rely on the model for medical, legal or financial decisions. Do not act on a fact it gives you without checking it. And do not use it for calculations you need to be exact, because its arithmetic is weak.

How fast it runs

Speed depends far more on the device than on the model’s label. In headless Chrome 155 on an AMD Ryzen 5 7530U laptop, the English model produced 21.9 tokens per second on its first reply and 18.1 on its second, using six threads. The phone rows in the table are emulations: a laptop was limited to approximate phone cores and memory. They are rough guides, not measurements on real phones. The emulations show how the engine behaves with fewer cores and less memory, which is useful, but a real phone may differ in both directions.

The live public site gave one more data point. A real English reply arrived after 71.5 seconds, including the 571 MB download, at 25.7 tokens per second, with peak tab memory of 1,286 MB. That was a single run, so treat it as one data point rather than a benchmark. A slow phone waits far longer for the first reply, and long conversations get slower as they grow.

Getting the best from it

Short, specific prompts work best. Ask for one thing at a time, paste in the text the model should work on rather than expecting it to remember, and say what form you want the answer in. The guide to clear prompts in English and Turkish has examples in both languages.

Then check the facts. Names, numbers, dates and quotations are where small models most often go wrong, so verify them against a source you trust. The article on checking AI answers shows how to test a reply without starting over.

When a task needs broad knowledge, long reasoning, exact calculation or current information, reach for a larger model or a tool that shows its sources. A small model is a useful first draft and a reasonable helper for simple jobs. It is not a substitute for the tools that do harder jobs better, and it should not be asked to be one.

One download, checked file by file

Your first message downloads the model for the language you chose, about 550 megabytes split into small parts. Each part has a fingerprint (a SHA-256 hash) published in a small manifest; the app refuses any part that does not match. Verified parts go into your browser’s cache, so the next visit can start without downloading again, unless the browser clears its storage.

A model shrunk on purpose

Each model is a 0.6-billion-parameter-class network adapted from Qwen3-0.6B. Two changes make it small enough for a tab. Its vocabulary keeps only the word pieces its language actually uses, and its numbers are stored as 8-bit integers instead of 16-bit floats, which halves the download without a measurable change in its answers. We also tried 4-bit versions, half the size again, but every one we measured answered worse, so the app does not use them. Every layer can look back over the whole conversation: limiting most layers to recent text made long chats worse.

A small engine written in C

The calculation runs in WebAssembly, compiled from about a thousand lines of C written for this model. It uses the processor’s vector instructions, can split the work across several processor cores, and keeps the fixed instructions at the start of every chat pre-computed, so your first reply does not wait for them. Long conversations stay manageable because older text is stored compactly.

Honest about being small

A model this size knows much less than large assistants. Flyingkiwi was trained to answer plainly when it knows, to say when it does not or cannot know, such as today’s news, your own private details or the future, and to correct questions built on a false assumption. It still makes mistakes, so check anything that matters. The technical report lists what was measured, including where it falls short.

1. Find the claim you would act on

Suppose an assistant helps you plan a visit to a local museum. The suggested order of exhibits is an idea. “The museum closes at 18:00 on Sunday” is a factual claim. Check the opening time on the museum’s own website before relying on the plan.

2. Separate your information from its assumptions

Ask: “Which details came from my message, and which did you assume?” If you provided a timetable, a good answer should use that timetable. If you did not provide your travel time, an estimate should be labelled as an estimate. This makes the answer easier to inspect; it does not prove that the model’s explanation is correct.

3. Check the source, not just the presence of a link

Open the source and look for the exact detail. Does it refer to the right place, date, and situation? A real link to a museum homepage does not necessarily support a claim about a holiday schedule. When no source is available, treat the claim as unverified.

4. Try one meaningful follow-up

Change a constraint: “I can only arrive after 16:00.” Does the reply preserve the details that still apply and revise the parts that changed? Repeating the original answer in a friendlier tone is not the same as adapting the plan.

5. Keep a place for “not enough information”

A useful response may ask which museum you mean or say that it cannot verify current hours. Check whether the uncertainty matches what is missing. Avoid rewarding an assistant simply for refusing every question: an answerable question still deserves a useful answer.

Read accuracy alongside coverage

An accuracy percentage needs a denominator. A system that answers fewer questions can show a higher rate without getting more questions right. Look at correct answers, attempted answers, and unanswered questions together.

Flyingkiwi’s research report retains the historical counts and the 24 October correction to its comparison settings.

Give the task before the background

“Help me write a short invitation” tells the assistant what to do. Follow with the event details and your audience. Long background without a clear task leaves more room for an irrelevant response.

Provide the details you want preserved

For an imaginary book swap:

Write a friendly invitation to our neighbourhood book swap. It is on Saturday, 7 November, from 14:00 to 16:00 in the community room. Ask people to bring up to three books. Keep it under 70 words and do not add details I have not provided.

The same task in Turkish:

Mahallemizdeki kitap takası için samimi bir davet yaz. Etkinlik 7 Kasım Cumartesi, 14.00–16.00 arasında ortak salonda. Katılımcılardan en fazla üç kitap getirmelerini iste. 70 kelimeyi geçmesin ve vermediğim ayrıntıları ekleme.

Make the follow-up specific

Instead of “make it better,” try “Keep the time and book limit, but make the tone less formal.” That gives you a concrete check: did the revision retain both facts? If you switch languages, name anything that must remain exact, such as a person’s name or a venue title.

Use a short source when the answer should stay grounded

For a summary, paste the passage and say whether outside information is allowed. For a comparison, state the criteria. For an explanation, describe what you already know. You can ask the assistant to flag missing information, then check whether it actually does so.

Clear wording has limits

A careful prompt cannot supply knowledge the model lacks or establish that a fluent answer is true. Verify names, numbers, and external facts when they matter. Our answer-checking guide offers a short way to do that.

Explore Flyingkiwi’s model design and training record.

The first visit can include a model download

A model is more than the chat page you see. Its weights must be available to the runtime before it can generate text. A prepared site can show the download size and reuse cached files on later visits. That can reduce repeated downloads, but it should not promise that the browser will retain them forever.

Browsers set their own storage quotas and can remove data under storage pressure. Persistent-storage requests may help when granted, but users can still clear site data. MDN explains these differences in its storage and eviction guide.

A device check is only a first step

WebGPU gives a supporting browser access to GPU computation. Its availability depends on the browser and environment, and it requires a secure context. See the WebGPU documentation. A successful capability check does not establish that a particular model will fit in memory, export correctly, or return useful answers.

Look at the whole privacy path

A local model can avoid sending your prompt to a remote inference service. Separately, page requests can still reach a hosting provider, and analytics can still exist. Read the site’s privacy notice instead of treating “local” as a complete privacy statement. Downloading a model also does not mean every part of the website works offline.

What this means for Flyingkiwi

Flyingkiwi’s chat now works this way: the selected language’s model downloads once from this site, every file is checked, and replies are generated on your device by a WebAssembly engine. See the privacy notice for storage and hosting details, or the technical area for how the models were built and evaluated.