Flyingkiwi: bounded English and Turkish model adaptation

1. Summary

Flyingkiwi explores separate English and Turkish conversational models derived from Qwen3-0.6B. Both runs completed 81 optimizer updates on an NVIDIA A100-SXM4-40GB and passed the saved-checkpoint loss check. Development negative log-likelihood (NLL) decreased in both languages.

The English and Turkish response collections each cover 100 cases and 120 turns per model. Their historical counts are retained below, with the decoding mismatch identified separately from the training measurements.

2. Method

Reference and architecture

The reference is Qwen/Qwen3-0.6B at revision c1899de. The candidate retains eight key/value heads, widens the feed-forward intermediate dimension to 3,584, and configures a 1,024-token sliding window with a global attention layer every fourth layer and at the final layer. The resulting model has 640,090,112 parameters.

Existing projection weights are copied. New feed-forward input rows retain seeded initialization and their corresponding output columns start at zero. Retaining the original KV head count avoids pooling heads in these runs. At the 512-token training length, the local window does not remove earlier tokens. These construction choices do not establish a quality gain or browser compatibility; numerical equivalence and longer-context behavior require separate checks.

Data and split boundaries

Each language has 322 training prefixes from 305 conversations in 205 source or template families, 32 development cases, and 100 frozen evaluation cases. English has 10,452 supervised training tokens and 420 development target tokens; Turkish has 14,074 and 585 respectively. Public source trees, translations, and exercise families are kept in a single partition to reduce direct overlap across splits.

The corpus combines retained or adapted public-source material with assistant-authored teaching examples. In each language, 26 training prefixes carry Apache-2.0 source terms, 73 carry Dolly-derived CC BY-SA 3.0 terms, and 223 are labelled project-authored synthetic text. These are prefix counts, not counts of independent conversations. The corpus does not use private user chat.

Completion-only objective

The tokenizer’s chat format supplies the system and conversation context. Only the final assistant completion and its end-of-sequence token contribute to the causal language-model loss. Earlier turns and padding receive the ignore label. No audited training or development example is truncated at the 512-position limit.

L = −(1 / N) Σ log pθ(yₜ | context, y<ₜ)

Here, N counts supervised completion tokens. Up to four single-example microbatches form one update; each loss is weighted by its supervised-token count within that update. This is supervised adaptation, not reinforcement learning, distillation, or training from scratch.

Optimization and checkpoint selection

All parameters are trainable, with FP32 live parameters and BF16 autocast. Training uses AdamW with weight decay 0.01, gradient norm clipping at 1.0, gradient checkpointing, seed 43, and one shuffled pass capped at 81 updates. Examples are padded to 512 positions during training. The learning-rate schedule uses approximately 10% warmup and cosine decay to 10% of its peak. The effective peak is 1 × 10⁻⁶; the optimizer’s initial 2 × 10⁻⁶ value is replaced before each update.

Development NLL is checked before training, every 25 updates, and at the final update. An improvement greater than 0.001 can select a checkpoint; development loss more than 2% above the initial value triggers a stop. English selected update 81 and Turkish selected update 75. Selected snapshots are stored in BF16 and reloaded for a separate development loss check. Optimizer state and FP32 master parameters are not saved, so the artifacts are not complete resumable training-state snapshots.

Generation evaluation

The earlier evaluators intended matched greedy decoding with thinking disabled and at most 128 new tokens per turn. They set do_sample=False only on a supplied generation configuration and omitted use_model_defaults. In Transformers 4.57.1, a value matching the global default can be treated as unset and replaced by a saved model default. The reference-decoding warning explicitly records restored do_sample=True.

The migrated candidate’s saved generation configuration has _from_model_config=true and omits do_sample, consistent with the global False default. Its decoding mode is inferred from that artifact rather than an explicit per-call record. The old reference and candidate settings therefore were not matched. This affects the sequential English comparison and the Turkish collection attempts; the original manifests’ matched “greedy” labels are superseded. The candidate’s reviewed outputs remain preliminary whole-response observations, not a controlled comparative result.

The official Transformers 4.57.1 API documentation explains how model defaults are inherited and how direct generation arguments override configuration values. The corrected evaluator explicitly passes use_model_defaults=False, do_sample=False, and num_beams=1. A tiny CPU contract test reproduces the reference-style sampling override, confirms the corrected effective settings, and repeats the same tokens across different seeds. This checks the decoding contract only; it is not a language-quality benchmark. No corrected matched greedy frozen comparison has completed at this report revision.

Input plus the 128-token generation allowance must fit within 1,024 tokens; prompts are not silently trimmed. Multi-turn runs use the model’s own generated history rather than target answers. The earlier sequential Turkish attempt had a 450-second bound; the later completed batch-eight job had a separate 393-second maximum. The historical English user-priorities-v2 review is a post-hoc assistant reassessment of an authored test, not independent human or native-speaker evaluation. Coding remains a secondary diagnostic.

3. Measured results

Times cover the training stage. Reloading the selected BF16 checkpoints preserved the development loss improvement.

Historical English response observations

The original review compared sampled reference responses with candidate responses whose settings are inferred as greedy. These counts describe those outputs; the decoding settings differ.

Both models have 21 correct primary factual answers; the candidate’s one fewer direct answer changes the denominator from 34 to 33. Four cases are correct only in each model’s output, and nine are incorrect or incomplete in both. The historical coding counts are 2/3 and 1/3. Different decoding settings prevent attributing these differences to training.

Turkish: completed collection with mismatched settings

The later Turkish collection completed 100 cases and 120 turns per role in batches of eight. It took 200.47 seconds for the job and 175.838 seconds for the evaluation stage. Reference sampling and inferred greedy candidate settings were mismatched.

The earlier 450-second attempt reached 96 reference cases, 112 turns, and 11,385 generated tokens, with zero candidate cases. Both attempts are retained in the aggregate record.

A subsequent whole-response review of the saved Turkish candidate found 8 correct answers among 25 direct factual answers, with 25/34 answer coverage, 1/20 appropriate uncertainty responses, and 3/18 multi-turn cases after excluding exact-format diagnostics. Nine unresponsive factual cases remain missing coverage. Coding was a secondary diagnostic and passed 3/3. These small assistant-reviewed observations describe the candidate; they do not establish a matched training improvement or independent native-speaker assessment.

4. Interpretation and limitations

Lower development NLL is the clearest training result. A matched generation comparison is needed to assess changes in response quality.

• Decoding protocol mismatch. Reference and candidate generation settings differed; the historical counts cannot isolate training effects.
• Small and correlated data. In each language, 120 exercise variants share only 20 templates. English later-turn supervision appears in 17/322 prefixes and contributes 495 target tokens. The corpus offers limited evidence of broad conversation coverage.
• Development-set selection. The English development target contains only 420 tokens. It uses teacher-forced context; real rollouts can compound earlier mistakes. Repeated selection on this set can overstate general progress.
• Subjective and post-hoc scoring. The test and review are assistant authored. Some rubric judgments are subjective, the English priorities were reassessed after generation, and pretraining exposure is unknown. The counts describe these cases only.
• No validated Turkish quality comparison. A complete response collection exists with mismatched settings, but no corrected matched greedy comparison or independent native-speaker validation is available.
• No browser release evidence. These metrics do not establish trained ONNX export parity, quantized quality, cache correctness for released weights, broad device compatibility, long-context quality, or production safety.

Next research questions

Next experiments can broaden conversation families, add varied later-turn revisions, rebalance repeated templates, and enlarge development coverage. The short-context migration control does not replace broader language and long-context evaluation. Dynamic padding remains an efficiency hypothesis; its tiny CPU loss/gradient check has not established a GPU speedup.

5. Provenance and licensing

The reference model’s official model card identifies Apache-2.0. Preserve its license and applicable notices in any derived release. This model license does not replace the terms of source datasets.

• OpenAssistant/oasst2: OpenAssistant contributors; Apache-2.0. Selected English prompts and assistant-authored corrections or Turkish adaptations are recorded as modifications.
• databricks-dolly-15k: copyright 2023 Databricks, Inc., with underlying quoted Wikipedia material attributed to its editors and contributors; CC BY-SA 3.0. Retained text and Turkish adaptations keep attribution, change notices, and applicable share-alike obligations. The source provider does not give individual Wikipedia revision URLs for these rows; none are invented here.
• Original teaching and evaluation text is labelled project-authored synthetic text. It is not presented as human conversation or independent review.

Code, derived weights, and adapted data require separate license notices. Source terms and modification records must accompany any redistribution; this report assigns no new license to the weights.

6. Evidence record

Download the aggregate record (JSON): training measurements, historical response counts, decoding qualifications, and source-file hashes.

Version 3 adds the Turkish candidate review and short-context development controls. Version 2 corrected the generation labels. The aggregate record retains the original observations, source hashes and decoding qualifications.