8-bit vs 4-bit weights: what quantisation costs a small model

Every language model is a large collection of numbers, and the format used to store those numbers decides how big the download is and how far the answers can drift. Flyingkiwi’s English model would be 341 MB instead of 561 MB if its weights were stored in 4 bits. It ships in 8 bits. This article explains the measurements behind that choice, including one result that looked wrong until we checked how the model had been trained.

The trade: download size against answer quality

Making the weights smaller is the most direct way to make a model faster to download. The cost is that every weight becomes a rougher approximation of the value training gave it. Storing weights with fewer bits is called quantisation, and the useful question is always how much answer quality each saving costs.

For English, the options were 8-bit weights at 561 MB, plain 4-bit weights at 341 MB, and several mixed versions in between. In English, only the 8-bit version matched the unquantised checkpoint to the three decimal places we report. The rest of this article shows what the smaller files lost, and why one number from one test is not enough to choose between them.

What quantisation does, concretely

A 16-bit float can represent a very wide range of values. A quantised weight is stored as a small integer, and a scale turns that integer back into an approximate value. Flyingkiwi’s browser weights store 8-bit integers with one 16-bit float scale for every 32 values, so each scale is shared by a small group of neighbouring weights rather than by a whole matrix.

The difference between 8 and 4 bits is the number of steps available. An 8-bit integer has 256 levels, and a 4-bit integer has only 16. Imagine two rulers covering the same length, one marked in 256 steps and one in 16. Whatever you measure is rounded to the nearest mark, so the coarser ruler leaves larger errors. A model multiplies hundreds of millions of these rounded values, and small errors in many weights can add up to a different answer.

How the models were measured

The main yardstick is held-out answer loss, measured in nats per token. A held-out conversation is one the model did not train on. The loss tells you how well the model predicts the words of the assistant’s reply, so a lower number is better. The table below uses 40 English and 24 Turkish validation conversations.

Why only the assistant’s answers? A chat model is used to write the assistant’s turns, so the loss on those turns is closer to what a reader experiences. A general perplexity score averages over all kinds of text, including text the model is never asked to write. That average can hide a change in exactly the part of the conversation that matters.

The validation set is small. Treat the decimal places as a guide to direction, and rely on the blind tests further down to show whether a change reaches the answers.

Reading the table

In English, 8-bit storage cost 0.0003 nats per token. The float checkpoint and the shipped file both score 1.327, so at this precision the rounding made no visible difference. Turkish moved from 1.562 to 1.565 in the same comparison. The full set of measurements is in the technical report.

Plain 4-bit rounding cost 0.17 nats per token in English, and the Turkish loss rose from 1.565 to 1.804. GPTQ, the method described by Frantar et al. (ICLR 2023), roughly halved that cost in English, bringing the 4-bit English loss to 1.407. It quantises one layer at a time, column by column. After each column is rounded, the method pushes that rounding error onto the columns not yet quantised, using statistics from the layer’s inputs to decide where the error goes.

Keeping the most sensitive matrices in 8 bits brought the 420 MB English model to a loss of 1.365. It is still more than a hundred times the cost of 8-bit weights, at 0.039 nats per token. Sensitivity was ranked with the diagonal Fisher information of the answer loss, and the matrices kept in 8 bits were mostly the small value and key projections.

Does loss predict answers?

Loss is a proxy. To check whether a change matters, we ran blind tests on 100 project-written cases per language. Answers from two models were mixed, and an AI grader compared each answer with a reference answer without knowing which model had written it. The grader counted correct answers and, separately, replies that contained invented details.

The English models with 4-bit weights, including the mixed version, lost between 4 and 9 points. In each of those rows the number of invented-detail flags went up. The shipped 8-bit model scored the same as the unchanged reference, with the same number of flags.

The reference total moves by a few points between runs, from 62 to 66 in English, so differences of two or three points are within grader noise. Each category has 10 to 30 cases, and one case moves a category score by 3 to 10 points. The shipped variant was chosen with this same test, so its score is slightly optimistic.

The QAT trap

The most useful lesson came from the Turkish v3 model, and it nearly led us astray. On that model, plain 4-bit rounding looked almost free: 1.5425 nats per token against 1.5401 at 8 bits. GPTQ looked harmful, at 1.5965. Someone who saw only those numbers might conclude that plain rounding is the better method.

The explanation is how that checkpoint was trained. The v3 Turkish model had been trained with 4-bit quantisation-aware training on exactly that rounding grid. Plain rounding therefore reproduced the model that training had produced, while GPTQ moved the weights away from that grid. The comparison was measuring how well each method matched a grid the model had already learned, not which method preserves an ordinary model best.

Without quantisation-aware training, as in v3.1, the picture reverses. Plain 4-bit rounding raised the Turkish loss from 1.565 to 1.804, and GPTQ raised it to 1.664. The rule is simple: check how a checkpoint was trained before reading quantisation results. A result on a model trained with QAT says something about that training, and little about rounding an ordinary checkpoint. How the v3.1 models were trained instead is explained in the knowledge distillation article.

What would change the decision

v3.1 ships 8-bit weights, so the English browser download is 571 MB including the engine, tokenizer and prompt cache. The 561 MB of weights is the largest part of that download. The engine that reads these files is described in the WebAssembly article, and it already supports 4-bit weights.

The open route is quantisation-aware training of v3.1 on its own 4-bit grid. That has not been tested, so the 341 MB figure is not a result for this model. The open question is the trained model, not the code. Until a version of that kind passes blind tests against the unchanged reference, 8 bits remain the release.

A checklist for quantising a small model

If you are quantising a small model yourself, these steps address each problem described above.

1. Find out how the checkpoint was trained. A model trained with QAT can make plain rounding look better than it is for an ordinary checkpoint.
2. Measure on the assistant’s answers, not only on general text.
3. Measure the format you will actually ship, with its real scales and group size.
4. Compare against the unchanged model at the same settings.
5. Run blind tests before release, and count invented details separately from correct answers.

The blind-test method is covered in more detail in how to evaluate a chatbot honestly. The conclusions here belong to one 0.6B model and one set of cases, so treat them as a worked example rather than a rule for every model.