Chapter 3 of 6

Making it small: 8-bit and 4-bit weights

Every weight is stored as a small whole number and a scale shared by its group. Fewer levels make a smaller download and a rougher copy of the model.

Fig. 1Detent rule

Drag or press ← → · Space swaps racks

4-bit · 16 levels · error 0.20 step

plunger, in the nearest notch
pointer, at the true value

Fig. 1 — Detent rule. Each weight snaps to the nearest of 256 (8-bit) or 16 (4-bit) levels of its group's scale (the carriage's positions are illustrative); for Flyingkiwi 0.6B, 8-bit costs almost nothing (answer loss 1.327 → 1.327) while plain 4-bit raised it to 1.496, so the 0.6B models ship at 8-bit.

In numbers

256 / 16

Levels at 8-bit and 4-bit

Every 32 weights share one 16-bit scale.

1.327 → 1.327

Answer loss, full precision → 8-bit

Flyingkiwi 0.6B English, held-out answers; lower is better.

1.496

Answer loss with plain 4-bit

GPTQ 4-bit reached 1.407, so the 0.6B models ship at 8-bit: 561 MB of English weights.

69.5 MB

Blazing Kiwi, 4.5 bits per weight

It was trained with 4-bit rounding simulated, so it ships at 4-bit.