Flyingkiwi v3.1: shorter answers, fewer invented details

1. Summary

Each language has its own model built from Qwen3-0.6B, with a vocabulary cut down to what the language uses, fine-tuned on answers from a much larger teacher and run on the visitor's device by a dependency-free C engine compiled to WebAssembly. Nothing is sent to a server.

The v3 release invented more details than the unchanged base model and the English model lost ground on TruthfulQA. v3.1 retrains both models on the same conversations after the teacher rewrote every answer into its shortest complete form. On the frozen 100-case tests, graded blind against unchanged Qwen3-0.6B, English v3.1 scored 63/100 against 63/100 with 6 invented-detail flags against 6 (v3: 11 against 6), and Turkish 37/100 against 12/100 with 17 flags against 19. English TruthfulQA is back to 25.1% against 25.8% for the base model (v3: 21.3%). Answers are shorter: 51 instead of 83 tokens on average in English, 92 instead of 256 in Turkish.

The models still download about 571 MB each: every 4-bit format we measured raised the held-out loss far more than 8 bits (section 4). Turkish remains weak in absolute terms.

2. Models

Unchanged from the reference: 28 layers, width 1,024, 16 query and 8 key/value heads of 128 dimensions, a 3,072-wide SwiGLU feed-forward block, QK-norm and RoPE, so the checkpoints load with stock Hugging Face Transformers (Flyingkiwi-EN-0.6B, Flyingkiwi-TR-0.6B). Changed:

• Language-specialised vocabulary. Token frequencies were counted on the language's training data, a web sample and code; rare tokens were removed and the byte-level BPE rebuilt, closed over merge ancestors. Text made only of kept tokens tokenises exactly as before, and the 256 byte symbols remain, so nothing becomes unencodable. The embedding and the output layer shrink by almost half.
• Full attention, trained that way. v3 was trained with a 1,024-token sliding window in three of every four layers and released with full attention, because full attention gave lower loss on long conversations (1.08 against 1.28 nats per token after the first 1,024 tokens). v3.1 trains with full attention from the start.
• Concise distillation. Section 3.
• 8-bit browser weights. Section 4.

3. Training

The teacher was Qwen3.5-35B-A3B (Apache-2.0, FP8) answering under a written rule set: be accurate, say when unsure or when something is unknowable, correct false premises, ask when a request is ambiguous, use Markdown and code blocks. The students learned from those answers with the short system prompt the app sends.

• Prompts. English: 15,834 training and 344 validation conversations: 3,369 human-written prompts (OASST2, Aya), 653 GSM8K word problems, 6,292 teacher-written everyday, explanation, writing, coding and maths requests, 1,890 honesty probes, 479 questions about invented entities, 125 identity questions, 248 memory checks and 1,198 chained conversations; 1,544 continue with teacher-written follow-up turns and 2,385 have no system prompt. Turkish: 20,778 training and 388 validation conversations: 7,761 human-written prompts (OASST2, Aya, translated OASST2), 433 GSM8K word problems, 7,455 teacher-written everyday, explanation, writing, coding and maths requests, 1,941 honesty probes, 460 questions about invented entities, 118 identity questions, 474 memory checks and 1,567 chained conversations; 2,393 continue with teacher-written follow-up turns and 3,230 have no system prompt.
• Cleaning. Before v3's second training run, answers with unprompted remarks about the assistant's own limits, thinking aloud, or maths results stated before the working were filtered or reordered (details in the v3 report).
• Concise rewrite (new in v3.1). v3's answers were two to four times as long as the base model's, and blind grading flagged more invented details: every extra sentence is another chance for a 0.6B model to state something false. Asking for brevity in the system prompt did not shorten them, so the teacher saw each whole conversation and rewrote every assistant message into the shortest complete answer, keeping code blocks, requested formats, honest statements of uncertainty and anything a later turn refers to. A rewrite that dropped a code block, added a disclaimer, grew longer or switched language kept the original. English: 22,067 answers rewritten, 43% of the characters kept; Turkish: 30,036 rewritten, 33% kept.
• Recipe. Full-parameter fine-tuning of Qwen3-0.6B (BF16 with FP32 master weights), two epochs, peak learning rate 1.5e-05 with cosine decay, no quantisation-aware training: English 148 steps (validation loss 2.165 → 1.311), Turkish 300 steps (2.610 → 1.632).
• Compute. One NVIDIA RTX PRO 6000 Blackwell (96 GB) on the existing Colab Pro+ plan: rewriting 57 minutes, training 7 (English) and 15 minutes (Turkish), about two hours of session time in total, inside a written three-hour limit. No purchase or plan change.

The frozen test sets were never uploaded or trained on; evaluation ran locally with the native build of the browser engine.

4. Why 8-bit weights

4-bit weights would take about 341 MB instead of 561 MB for English. We measured the cost on held-out validation conversations as the loss of the assistant tokens, with each format simulated exactly as the engine stores it (32 weights per fp16 scale; tools/quant_plan.py):

8 bits cost 0.0003 nats per token in English. Plain 4-bit rounding cost 0.17. GPTQ (Frantar et al., 2023), which pushes each rounding error onto columns not yet quantised using the layer's input statistics, halved that. Keeping the most sensitive matrices in 8 bits, ranked by the diagonal Fisher information of the answer loss (mostly the small value and key projections), brought a 420 MB model to 0.039. That is still more than a hundred times the 8-bit cost, and earlier blind tests of 4-bit models lost 4–9 points (section 5), so v3.1 ships 8 bits.

One trap is worth recording. On the v3 Turkish model plain 4-bit rounding looked almost free (1.5425 against 1.5401 nats at 8 bits), and GPTQ looked harmful (1.5965). That model had been trained with 4-bit quantisation-aware training on exactly that rounding grid, so plain rounding reproduced the model it was trained as, and GPTQ moved it away. Without that training, as in v3.1, the picture reverses.

5. What we tried

Each row is one blind grading run of the 100 English or Turkish cases, with the variant's answers mixed at random with the unchanged Qwen3-0.6B answers. The reference total moves by a few points between runs (62–66 in English), so differences of two or three points are within grader noise. The last column counts answers the grader flagged for invented facts, rules or claims.

Not in the table: a brevity instruction in the system prompt (no shorter answers), 4-bit quantisation-aware training (tried for v3; its 4-bit exports lost in blind tests, rows above), and a sliding attention window (higher loss on long conversations). The released variants were chosen with this same test, so their scores are slightly optimistic.

6. Browser engine

engine/flyingkiwi.c implements the decoder in about a thousand lines of C with no dependencies and compiles with clang straight to WebAssembly (no Emscripten); the same source builds natively for evaluation.

• Integer kernels. Weights are 8-bit integers with one scale per 32 values (4-bit is supported) and activations are quantised per 32 values, so matrix products use SIMD integer dot products. Where the browser supports relaxed SIMD, one instruction performs 16 multiply-adds with 7-bit activations.
• Cache. Keys and values are stored as 8-bit numbers with one scale per 32 values, which keeps Qwen3's outlier key channels accurate; the first four positions (attention sinks) stay in full precision. The fixed system prompt's cache ships with the model, and each new turn only processes what changed.
• Threads. The site is cross-origin isolated, so the engine shares memory between up to six worker threads; rows and attention heads are handed out through an atomic counter, which suits phones with fast and slow cores. Threaded results are bit-for-bit identical to single-threaded ones.
• Verification. The released files stay within a mean KL of 0.0114 (English) and 0.0513 (Turkish) of their checkpoints at the first reply token of 30 development prompts (WebAssembly relaxed-SIMD build: 0.0147 and 0.0563); every downloaded file is hash-checked before use.

7. Small devices

We ran the real chat page in a fresh headless Chrome with the CPU and memory of the whole browser capped: a mid-range phone profile (the page reports four cores, so the app uses two engine threads, each at about half a laptop core, 2 GB of memory) and a low-end profile (one thread at about a quarter of a laptop core, 1.5 GB). This is emulation on an AMD Ryzen 5 7530U laptop, not a phone: it does not reproduce ARM cores, heat throttling or mobile browsers.

Both models load and answer within 1.5 GB of memory. On the low-end profile replies come at about 2.4 and 2.0 tokens per second, slow but usable for short answers; devices that report 4 GB of memory or less get a 2,048-token context, which halves the cache.

8. Evaluation

Each language has a frozen set of 100 project-written cases that were never used for training: 30 answerable questions, 20 the model cannot know (live, private or future information), 10 false premises, 10 ambiguous requests, 10 friendly messages and 20 multi-turn follow-ups. Flyingkiwi ran as the released 8-bit file in the native build of the browser engine; the reference is unchanged Qwen3-0.6B (8-bit, KL 0.0014 from full precision). Both received the app's system prompt and greedy decoding with repetition penalty 1.1. Answers were graded blind by AI graders against reference answers, with spot checks by the assistant preparing this report; there was no independent human or native-speaker panel, and 10–30 cases per category means each count moves by 3–10 points per case.

English

Turkish

TruthfulQA MC1 (817 questions; each option scored as the assistant reply in the same chat format): Qwen3-0.6B 25.8%, Flyingkiwi English v3.1 25.1%, v3 21.3%. The remaining gap is within the test's sampling error (about 1.5 points).

Mean answer length on the frozen sets (tokens): English 51 (v3: 83, Qwen3-0.6B: 37), Turkish 92 (v3: 256, Qwen3-0.6B: 74).

English v3.1 matches the base model overall and in invented details while asking about ambiguous requests and admitting what it cannot know more often; it corrects false premises less often. Turkish v3.1 is far ahead of the base model, which mostly fails in Turkish, but 37/100 is not a reliable assistant: arithmetic and facts are often wrong.

9. Limits

• These are 0.6B-class models. They know far less than large assistants and can still state wrong things confidently, particularly niche facts, precise numbers, arithmetic and anything recent.
• They cannot browse, use tools, see files or remember earlier conversations, and they do not know today's date.
• Turkish quality was judged by an AI grader, not by native-speaker reviewers, and is low in absolute terms.
• The download is about 571 MB per language, kept in the browser's cache afterwards; speed depends heavily on the device, and long conversations get slower.
• The network architecture is Qwen3-0.6B's apart from the vocabulary: the changes here are in training data, weight format and the engine, not a new kind of model.

10. Files and licences

Models: https://huggingface.co/yunusemrejr/Flyingkiwi-EN-0.6B and https://huggingface.co/yunusemrejr/Flyingkiwi-TR-0.6B (Apache-2.0, derived from Qwen/Qwen3-0.6B). Teacher outputs come from Qwen3.5-35B-A3B (Apache-2.0). Prompt sources: OpenAssistant OASST2 and CohereLabs Aya (Apache-2.0), GSM8K (MIT) and TriviaQA questions. FineWeb-Edu, FineWeb-2 and github-code-clean were used only to count token frequencies.

Earlier reports are preserved: v3 release, 25 October 2026 and research runs, 24 October 2026. All numbers on this page are in results.json.

Türkçe özet

Her dilin kendi modeli var: Qwen3-0.6B’den, o dilin kullandığı belirteçlere indirgenmiş bir sözlükle ve çok daha büyük bir öğretmen modelin yanıtlarıyla eğitildi. Bağımlılıksız bir C motoru WebAssembly’ye derlenip modeli ziyaretçinin cihazında çalıştırır; hiçbir şey sunucuya gönderilmez.

v3.1’de öğretmen model, eğitim yanıtlarının her birini en kısa eksiksiz hâline getirdi (Türkçe yanıtlar karakterlerinin %33 kadarını korudu) ve iki model bu yanıtlarla yeniden eğitildi. Türkçe model, 100 vakalık donmuş testte değiştirilmemiş Qwen3-0.6B’nin 12/100 sonucuna karşı 37/100 aldı ve uydurma ayrıntı içeren yanıt sayısı 17 oldu (Qwen3-0.6B: 19). Yanıtlar ortalama 92 belirteç (v3: 256). Mutlak olarak hâlâ zayıf: aritmetik ve olgular sık sık yanlış olabilir.

4 bitlik ağırlıklar indirmeyi yaklaşık 340 MB’a düşürürdü, ancak ölçülen her 4 bit biçimi (düz yuvarlama, GPTQ, karma 4/8 bit) kaybı 8 bitten çok daha fazla artırdı; bu yüzden tarayıcı 8 bit ağırlık kullanır. Düşük donanımlı telefon benzetiminde (tek iş parçacığı, 1,5 GB bellek) model çalıştı, yanıtlar saniyede yaklaşık 2 belirteçle geldi.

Türkçe değerlendirme

Bu küçük ve yapay zekâ destekli bir değerlendirmedir; ana dili Türkçe olan bağımsız değerlendiriciler kullanılmadı. Model küçük olduğu için hata yapabilir; önemli bilgileri doğrula.