Flyingkiwi v3: two small models that run in your browser

1. Summary

Flyingkiwi now answers in the browser. Each language has its own model built from Qwen3-0.6B: a vocabulary cut down to what that language uses, fine-tuning on answers from a much larger teacher that admit uncertainty, and a dependency-free C engine compiled to WebAssembly that runs 8-bit weights on the visitor's device. Nothing is sent to a server.

On the project's frozen 100-case tests, graded blind against unchanged Qwen3-0.6B, the English model scored 65/100 against 62/100 and the Turkish model 32/100 against 12/100. Most of the gain is in saying when it cannot know something and in asking about ambiguous requests. Both models still state more invented details than the reference, and the English model scores lower on TruthfulQA (21.3% against 25.8%; section 6). Several things we tried made the models worse, and section 4 lists them.

2. Models

Unchanged from the reference: 28 layers, width 1,024, 16 query and 8 key/value heads of 128 dimensions, a 3,072-wide SwiGLU feed-forward block, QK-norm and RoPE. Changed:

• Language-specialised vocabulary. Token frequencies were counted on the language's training data, a web sample and code; rare tokens were removed and the byte-level BPE rebuilt, closed over merge ancestors. Any text made only of kept tokens tokenises exactly as before, and the 256 byte symbols remain, so nothing becomes unencodable. This shrinks the embedding and the work of producing each token's probabilities.
• Attention. The models were trained with sliding-window attention (the last 1,024 tokens) in three of every four layers, to bound memory on phones. On long held-out English conversations, full attention gave clearly lower loss after the first 1,024 tokens: 1.08 against 1.28 nats per token for the fine-tune, 1.17 against 1.54 for the English release. The releases therefore use full attention in every layer up to the app's 4,096-token context, which costs roughly 150 MB of extra cache memory in the longest conversations.
• 8-bit weights. Every matrix is stored as 8-bit integers with one fp16 scale per 32 values. The last 30% of training used 4-bit quantisation-aware training, but the 4-bit English export scored 57 against 66 and 8-bit attention with a 4-bit feed-forward block 58 against 65, while the same weights in 8 bits scored 66 against 65. The 8-bit download is about twice as large; the answers were worth it.
• English weight blend. The English release averages the fine-tuned weights with the original Qwen3-0.6B weights, 50/50 (WiSE-FT, Wortsman et al., 2022). The fine-tune alone wrote longer answers with more invented detail; the blend kept most of its gains in honesty and clarifying questions, wrote shorter answers and scored slightly higher (section 4). In Turkish the original model is weak and the blend scored only 18 against 11, so the Turkish release is the fine-tune alone.

The checkpoints are standard Qwen3ForCausalLM models and load with stock Hugging Face Transformers: Flyingkiwi-EN-0.6B and Flyingkiwi-TR-0.6B.

3. Training

The teacher was Qwen3.5-35B-A3B (Apache-2.0, FP8) answering under a written rule set: be accurate, say when unsure or when something is unknowable, correct false premises, ask when a request is ambiguous, keep answers concise, use Markdown and code blocks. The students learned from those answers with the short system prompt the app sends, so the behaviour does not depend on the long teacher instructions.

• Prompts. English: 15,834 training and 344 validation conversations: 3,369 human-written prompts (OASST2, Aya), 653 GSM8K word problems, 6,292 teacher-written everyday, explanation, writing, coding and maths requests, 1,890 honesty probes, 479 questions about invented entities, 125 identity questions, 248 memory checks and 1,198 chained conversations; 1,544 continue with teacher-written follow-up turns and 2,385 have no system prompt. Turkish: 20,778 training and 388 validation conversations: 7,761 human-written prompts (OASST2, Aya, translated OASST2), 433 GSM8K word problems, 7,455 teacher-written everyday, explanation, writing, coding and maths requests, 1,941 honesty probes, 460 questions about invented entities, 118 identity questions, 474 memory checks and 1,567 chained conversations; 2,393 continue with teacher-written follow-up turns and 3,230 have no system prompt.
• Cleaning. The first students (v3a) scored 58 against 64 in English (4-bit): the teacher had been told what the app cannot do and mentioned it in about a fifth of all answers, so the students learned to refuse tasks the conversation itself made possible, and they copied answers that state a result before working it out. Before the second training run every conversation was filtered. English: 14,980 conversations kept; 7,989 dropped for unprompted remarks about the assistant's own limitations, 999 for thinking aloud, 77 maths answers for opening with a wrong number; 608 maths answers reordered to give the working first, 243 memory tests cleaned. Turkish: 19,599 conversations kept; 1,014 dropped for unprompted remarks about the assistant's own limitations, 21 for thinking aloud, 656 maths answers for opening with a wrong number; 403 maths answers reordered to give the working first, 307 memory tests cleaned.
• Calibration. TriviaQA questions were put to the unchanged Qwen3-0.6B (greedy plus four samples) in each language. English: 498 questions the base model reliably answered keep a short correct answer, 1,426 it reliably missed get an honest “not sure” reply; Turkish: 260 questions the base model reliably answered keep a short correct answer, 697 it reliably missed get an honest “not sure” reply. Questions in between were left out.
• Recipe. Full fine-tuning in BF16 autocast with FP32 master weights, AdamW (β 0.9/0.95), peak learning rate 1.5e-05 with warm-up and cosine decay to 10%, gradient clipping 1.0, loss on assistant tokens only, 4-bit quantisation-aware training for the final 30% of steps. English 240 steps, 10.3M assistant tokens, 13 min, validation loss 2.153 → 1.395; Turkish 628 steps, 30.3M assistant tokens, 39 min, validation loss 3.083 → 1.549.
• Compute. One NVIDIA RTX PRO 6000 Blackwell (96 GB) on the existing Colab Pro+ plan: 5.3 h in total, of which 2.6 h of teacher generation; the rest probing, vocabulary building, training and packaging. Evaluation ran locally on a CPU.

The frozen test sets were never uploaded or trained on; evaluation ran locally.

4. What we tried

Each row is one blind grading run of the 100 English or Turkish cases, with the variant's answers mixed at random with the unchanged Qwen3-0.6B answers. The reference total moves by a few points between runs (62–66 in English), so differences of two or three points are within grader noise. The fabrication column counts answers the grader flagged for invented facts, rules or claims.

The released variants were chosen with this same test, so their scores are slightly optimistic.

5. Browser engine

engine/flyingkiwi.c implements the decoder in about a thousand lines of C with no dependencies and compiles with clang straight to WebAssembly (no Emscripten); the same source builds natively for evaluation.

• Integer kernels. Weights are 8-bit integers (4-bit is supported too) and activations are quantised to 8 bits per 32 values, so matrix products use SIMD integer dot products. Where the browser supports relaxed SIMD, one instruction performs 16 multiply-adds with 7-bit activations; against Transformers this measured a mean KL of 0.015 in English and 0.034 in Turkish, against 0.012 and 0.039 for the plain SIMD build.
• Cache. Keys and values are stored as 8-bit numbers with one scale per 32 values, which keeps Qwen3's outlier key channels accurate; the first four positions (attention sinks) stay in full precision. The fixed system prompt's cache ships with the model, and each new turn only processes what changed.
• Threads. The site is cross-origin isolated, so the engine can share memory between up to six worker threads; rows and attention heads are handed out through an atomic counter, which suits phones with fast and slow cores. Threaded results are bit-for-bit identical to single-threaded ones.
• Verification. With full-precision weights and cache the engine's logits match Transformers to 1e-4. The released 8-bit files stay within a mean KL of 0.0128 (English) and 0.0407 (Turkish) of their checkpoints at the first reply token of 30 development prompts; every file is hash-checked before use.

Speed on a laptop with an AMD Ryzen 5 7530U (6 cores) in Chrome 155: the browser engine generated 23.5 tokens/s in English and 21.9 in Turkish with six threads (relaxed SIMD, 8-bit weights), and 13.9 / 12.9 tokens/s on one thread, which pages without cross-origin isolation fall back to. The native build of the same engine reached 34 tokens/s with six threads. The model is downloaded once (about 570 MB); served from the same machine, loading it and writing the first reply took 18 s in English and 27 s in Turkish.

6. Evaluation

Each language has a frozen set of 100 project-written cases that were never used for training: 30 answerable questions, 20 the model cannot know (live, private or future information), 10 false premises, 10 ambiguous requests, 10 friendly messages and 20 multi-turn follow-ups. Flyingkiwi ran as the released 8-bit file in the native build of the browser engine; the reference is unchanged Qwen3-0.6B (8-bit, KL 0.0014 from full precision). Both received the app's system prompt and greedy decoding with repetition penalty 1.1. Answers were graded blind (the grader did not know which model wrote which answer) against reference answers by AI graders, with spot checks by the assistant preparing this report. There was no independent human or native-speaker panel, and 10–30 cases per category means each count moves by 3–10 points per case.

English

Turkish

TruthfulQA MC1 (817 questions; each option scored as the assistant reply in the same chat format): Qwen3-0.6B 25.8%, Flyingkiwi English 21.3%. This is a clear loss on a standard truthfulness test. The fine-tune before blending scored 21.8% and the first student, trained before the data cleaning, 26.1% (4-bit). Removing the teacher’s hedged answers probably cost this likelihood test, which rewards cautious options, but we have not verified that.

Both models received more fabrication flags than the reference, mostly for invented explanations around a correct core (a made-up grammar rule, a false etymology, an invented naming convention). Turkish is far weaker than English in absolute terms; the gain over the original model is real, but 32 of 100 is not a reliable assistant.

Gold answers: each frozen case's final assistant message. While preparing this evaluation we found that the rubric field used by the earlier review holds the first turn's answer for the 20 multi-turn cases and English text for the Turkish cases; those fields were not used. Both models were scored on the same 100 cases per language with identical prompts, system prompt and decoding (greedy, repetition penalty 1.1 over 256 tokens, at most 512 new tokens). The reference is Qwen3-0.6B with its own chat template and thinking disabled; Flyingkiwi uses its ChatML template. Evaluated the released 8-bit files (full attention) on all 100 cases per language: their outputs are identical to the graded runs of the same weights.

7. Limits

• These are 0.6B-class models. They know far less than large assistants and can still state wrong things confidently, particularly niche facts, precise numbers, arithmetic and anything recent.
• They cannot browse, use tools, see files or remember earlier conversations, and they do not know today's date.
• Turkish quality was judged by an AI grader, not by native-speaker reviewers, and is low in absolute terms.
• The download is about 550 MB per language, kept in the browser's cache afterwards; speed depends heavily on the device, and long conversations get slower.

8. Files and licences

Models: https://huggingface.co/yunusemrejr/Flyingkiwi-EN-0.6B and https://huggingface.co/yunusemrejr/Flyingkiwi-TR-0.6B (Apache-2.0, derived from Qwen/Qwen3-0.6B). Teacher outputs come from Qwen3.5-35B-A3B (Apache-2.0). Prompt sources: OpenAssistant OASST2 and CohereLabs Aya (Apache-2.0), GSM8K (MIT) and TriviaQA questions. FineWeb-Edu, FineWeb-2 and github-code-clean were used only to count token frequencies.

The earlier research report with the bounded 81-update runs and the decoding-protocol correction is preserved: technical report, 24 October 2026.

Türkçe özet

Flyingkiwi artık tarayıcıda yanıt veriyor. Her dilin kendi modeli var: Qwen3-0.6B’den, o dilin kullandığı belirteçlere indirgenmiş bir sözlükle ve çok daha büyük bir öğretmen modelin belirsizliği dürüstçe kabul eden yanıtlarıyla eğitildi. Bağımlılıksız bir C motoru WebAssembly’ye derlenip 8 bitlik ağırlıklarla modeli ziyaretçinin cihazında çalıştırır; hiçbir şey sunucuya gönderilmez.

Türkçe model, 100 vakalık donmuş Türkçe testte değiştirilmemiş Qwen3-0.6B’nin 12/100 sonucuna karşı 32/100 aldı; artış en çok bilinemeyecek soruları dürüstçe kabul etmekte. Mutlak olarak hâlâ zayıf: yanlış bilgi verebilir ve ayrıntı uydurabilir. 4 bitlik sürüm ve kayan pencere dikkati denendi, ölçümlerde kaliteyi düşürdükleri için yayımlanan sürümde kullanılmıyor.

Türkçe değerlendirme

Bu küçük ve yapay zekâ destekli bir değerlendirmedir; ana dili Türkçe olan bağımsız değerlendiriciler kullanılmadı. Model küçük olduğu için hata yapabilir; önemli bilgileri doğrula.