How to evaluate a chatbot honestly: blind tests and frozen sets

Most chatbot comparisons look convincing and prove very little. A blog post shows a handful of good answers from one model and a handful of weak ones from another. A leaderboard shows one number with no test attached. Neither tells you whether the second model would do as well on your questions, or whether the test was chosen to flatter one side. The method below avoids those problems. It is the one Flyingkiwi used to evaluate its own models, and it includes reporting the failures.

Why most comparisons mislead

The usual problems are easy to name. Examples are picked because they flatter one side. The test set changes from one run to the next, so nobody can repeat the result. The decoding settings, which control how the model picks each word, are not reported. The grader knows which model wrote which answer. And scores are given as percentages with no denominator, so a score from a handful of questions looks the same as one from a large set.

1. Freeze the test before training

Flyingkiwi’s test is a frozen test set of 100 project-written cases per language. The cases were never used for training, and the categories are fixed: 30 answerable questions, 20 the model cannot know because they concern live, private or future information, 10 false premises, 10 ambiguous requests, 10 friendly messages and 20 multi-turn follow-ups. Each category tests a different thing. Unknowable questions reward honesty, because the right answer is that the model does not know. Ambiguous requests reward a clarifying question. Multi-turn follow-ups check whether the model keeps track of the chat.

Keep the set frozen. Once you have studied its failures and changed the model to fix them, the set is no longer independent of your work. Flyingkiwi’s frozen sets were never uploaded or trained on, but the project still notes that choosing a release with the same test makes its score slightly optimistic.

2. Compare against an unchanged reference

A comparison needs a reference that you did not change. Flyingkiwi compares each release with unchanged Qwen3-0.6B, run under the same conditions: the same system prompt, greedy decoding and a repetition penalty of 1.1. The reference is the 8-bit file, which stays within a mean KL divergence of 0.0014 of the full-precision model. KL divergence is a measure of how far one model’s predictions drift from another’s, so a low value means the reference is a faithful copy and not a damaged one.

Then run the exact file you ship. A test on a full-precision model tells you about a different product. Flyingkiwi’s comparisons used the released 8-bit file in the native build of its engine. The native build uses the same source as the browser engine, but it is not the browser build itself, and that difference should be stated when you report results.

3. Grade blind

Mix the answers from both models at random, so the grader cannot tell which reply came from which model. This is blind evaluation. Flyingkiwi’s grading used AI graders that compared each answer with a reference answer, with spot checks. Count invented-detail flags separately from correctness. A reply can be mostly right and still contain a made-up name or number, and that is a different failure from a wrong answer.

Blind grading is only as good as the grader. Flyingkiwi’s graders were AI systems, and there was no independent human or native-speaker panel. Treat a Turkish score in particular as a rough signal rather than a verdict on fluency. The same caution applies to any test you run yourself, however carefully you hide the labels.

4. Know your noise

Run the same test twice and you will see different numbers. Flyingkiwi’s English reference moved between 62 and 66 across runs, so a difference of two or three points is within the noise. Categories have 10 to 30 cases, and each case moves a category score by 3–10 points. A single case changing its verdict can look like a trend.

Choosing a release with the same test makes its score slightly optimistic, because part of the choice was made by looking at that score. A public benchmark helps as a second check. The TruthfulQA paper by Lin, Hilton and Evans describes questions on which some people would answer falsely because of a common misconception. Its MC1 test has 817 questions, and its sampling error is about 1.5 points. It should supplement your own cases, not replace them, because it measures something narrower than your use.

5. Check your settings, not your intentions

Decoding settings change answers, and it is easy to believe you set them when you did not. In Hugging Face Transformers, greedy decoding set only in a generation config can be overridden by the model’s own sampling defaults. The safe pattern is to pass use_model_defaults=False, do_sample=False and num_beams=1 directly to the generate call, then check the effective values.

Flyingkiwi’s 24 October 2026 protocol correction is an example of this problem: the reference model’s answers turned out to have been sampled while the candidate’s were greedy, so the comparison was uneven. The historical counts were kept and labelled as preliminary observations from mismatched settings rather than hidden. The 24 October correction explains what changed.

6. Report the losers

A report that shows only winners tells you little. Flyingkiwi’s reports include rejected variants. An English fine-tune trained on unfiltered data scored 55 where the reference scored 65. A blend that kept only 25% of the fine-tune scored 53 against 64. Both were rejected. The Turkish model’s result is stated plainly as well: 37 of 100, against 12 for the base model, which is a clear gain and still a weak score.

The same discipline applies to size trade-offs. Blind tests of 4-bit models lost 4–9 points, so the v3.1 release ships 8-bit weights. The 8-bit and 4-bit weights article covers that trade-off in detail. Report the losers in your own tests too. If you publish only the variant that won, readers cannot tell whether the win was a result or a selection.

Limits of this method, and a checklist

This method has limits, and you should state them next to every number. The graders are AI systems. There is no native-speaker panel. Each set has 100 cases, and the release choice was made with the same test. None of this makes a comparison worthless. It means a score is a measurement with edges, not a verdict.

Before you publish a comparison of your own, check that you have:

1. a test set kept out of training, with any reuse for choosing stated;
2. categories that test different things, including unknowable questions, false premises and multi-turn context;
3. an unchanged reference run under the same system prompt and decoding settings;
4. the exact file you plan to ship, with its effective settings checked;
5. blind grading, with invented details counted separately from correctness;
6. a repeat run, so that small differences are not read as results;
7. every variant you tried, including the ones that lost, with its counts out of the total.