Running a language model in WebAssembly: SIMD, threads and memory

A language model writes one token at a time, and each token needs the model’s weights to be read again and multiplied against the current state. For a model of 0.6B class, that makes the work mostly a question of memory traffic and multiplication. This is why the format of the weights and the processor’s vector instructions matter so much to how quickly a reply appears. Flyingkiwi runs both of its language models in this way inside a browser tab, using a small engine written for that one job.

What one token costs

Every weight is read once for each token generated, and every read feeds a multiplication. A 0.6B model is therefore limited less by clever arithmetic than by how much data must move and how many multiplications can be done per step. A smaller weight format means less to read, and a vector instruction means several multiplications in a single step.

The site uses two inference worker scripts, so the calculation runs off the main page thread. The answers never leave the browser. Everything described below is what happens inside the tab after the model has been downloaded.

A small engine with no dependencies

The engine is about a thousand lines of C with no dependencies. It is compiled with clang straight to WebAssembly, without Emscripten, and the same source builds natively for evaluation on a laptop. That second point matters. Tests run the same arithmetic that the browser runs, so a measured result describes the code that ships rather than a separate reference implementation.

The engine runs on the CPU. It needs no GPU and no browser extension, but it does need a current browser with WebAssembly SIMD, such as recent Chrome, Edge, Firefox or Safari. Only the two inference worker scripts are allowed to compile WebAssembly, and the site’s Content Security Policy enforces that. Other scripts on the page cannot compile WebAssembly, which keeps the compiled code in one known place.

Integer kernels for 8-bit weights

The weights are 8-bit integers with one scale for every 32 values. The activations, the intermediate numbers that flow between layers, are also quantised, in groups of 32 values. That lets the core matrix products use integer arithmetic, which processors handle quickly, instead of floating-point arithmetic. The products use SIMD integer dot products from WebAssembly SIMD128.

Where the browser supports relaxed SIMD, one instruction, i32x4.relaxed_dot_i8x16_i7x16_add, performs 16 multiply-adds at once, with 7-bit activations. The relaxed SIMD proposal defines the instruction. Where that instruction is not available, the engine uses the ordinary SIMD128 path.

Prefill is the step that reads your whole prompt before the first reply token appears. During prefill, the engine unpacks each weight group once and reuses it for four tokens. Generation produces one token at a time, so it cannot share weights across tokens in the same way.

Threads and cross-origin isolation

The engine shares memory between up to six worker threads, and that needs SharedArrayBuffer. The MDN security requirements describe the condition: the browser makes SharedArrayBuffer available only to cross-origin isolated pages. The site therefore sends Cross-Origin-Opener-Policy as same-origin and Cross-Origin-Embedder-Policy as require-corp.

Work is handed out through an atomic counter. Matrix rows and attention heads go to whichever thread is free next, which suits a phone with a mix of fast and slow cores. Threaded results are bit-for-bit identical to single-threaded ones, so the thread count changes the speed of a reply but not its content.

The practical consequence is strict. Under cross-origin isolation, every resource the page loads must be same-origin or must opt in explicitly, for example with a suitable Cross-Origin-Resource-Policy header. The MDN page on Cross-Origin-Embedder-Policy sets out the rule. Anyone building a similar page should plan for it from the start, because a third-party resource without that opt-in is blocked. It is also why every file on this site, fonts included, is served from the site itself.

Memory: a compact cache and a fixed start

The attention cache stores keys and values as 8-bit numbers, with one scale for every 32 values. This keeps Qwen3’s outlier key channels accurate. The first four positions, known as attention sinks, stay in full precision, because the model relies on them heavily.

The fixed system prompt is cached ahead of time and shipped with the model as system.kv. Each new turn only processes what has changed since the previous one, so the first reply does not wait for the instructions to be read again. Long conversations still get slower, because there is more context for each new token to attend to.

The context window is 4,096 tokens. On devices that report 4 GB of memory or less, it is 2,048 tokens, which halves the cache. In the measured runs, peak tab memory ranged from 1,113 MB to 1,358 MB. The live public-site run peaked at 1,286 MB.

Getting the weights into the browser

The first message downloads the model for the chosen language. The full English download is 571 MB (571,378,688 bytes), including weights, engines, tokenizer and prompt cache. The Turkish download is 567 MB (566,790,616 bytes). The local AI article explains what that download means for a visitor.

The download is split into 17 parts for each language. Each part is checked by size and SHA-256 hash before it is used, and verified parts are kept in the browser’s Cache Storage. A later visit normally reuses them, unless the browser has cleared its storage.

Speed on a laptop and on emulated phones

The measurements came from headless Chrome 155 on an AMD Ryzen 5 7530U laptop. The phone rows are emulation. The processor was limited to about half or a quarter of a laptop core, and the memory was capped. They are not measurements on real phones, and real devices will differ.

On the public site, a real English reply arrived after 71.5 seconds, including the 571 MB download. It ran at 25.7 tokens per second with relaxed SIMD and six threads, and peak tab memory was 1,286 MB. That is one run on one device and one connection, so read it as an example rather than a benchmark. Speed depends heavily on the device, and the emulated low-end rows show how much a weaker processor changes the wait.

Is the browser build faithful?

A simple check of the browser build uses KL divergence, which measures how far one probability distribution is from another. Here it compares the engine’s distribution over the next token with the one from the checkpoint. A value of zero would mean the two are identical.

On 30 development prompts, measured at the first reply token, the released English file stays within a mean KL divergence of 0.0114 of its checkpoint. The WebAssembly relaxed-SIMD build measures 0.0147. Both values are small, but they are not zero, so the browser build is close to the checkpoint rather than identical to it. The Turkish figures are higher: 0.0513 for the released file and 0.0563 for the relaxed-SIMD build, so Turkish replies carry a little more drift.

Fidelity is only one check. A faithful engine can still run a model that gives weak answers, so the quality results in the technical report remain the measure of quality. The limits are plain: a 0.6B model knows little, long conversations slow down, the download is about 570 MB per language, and speed varies widely between devices.