Chapter 1 of 6

Inside the model: one token at a time

A language model writes one token, a word or a piece of one, at a time. Each new token climbs every layer, and in each layer it looks back at what that layer kept for the tokens before it.

Fig. 1The layer stack

Arrows or Space step through the layers

token 4 · layer 1 of 28 · reads 3 earlier tokens

token being read now
kept for this token (KV cache)
shelf not in this model

Fig. 1 — The layer stack. Each token climbs every layer (28 in Flyingkiwi 0.6B, 30 in Blazing Kiwi); in each layer it reads what that layer kept for the earlier tokens (the KV cache), and at the top one next token is chosen. The tokens and the candidate scores are illustrative.

In numbers

Layers in Flyingkiwi 0.6B

Blazing Kiwi has 30 narrower ones: 576 numbers wide against 1,024.

4,096

Tokens of context in the browser

2,048 on devices that report 4 GB of memory or less.

16 / 8

Query and key-value heads per layer

In Flyingkiwi 0.6B, 128 numbers each. Blazing Kiwi has 9 and 3 of 64.

85,121

Tokens in the English vocabulary

Turkish keeps 78,255; Blazing Kiwi 27,799 of its base model’s 49,152.