Why open datasets matter for AI, from small models to AGI

A dataset is only as useful as your ability to inspect it. “Open data” for AI means more than a download link. The data has to be obtainable, its licence has to allow your use, it has to be documented well enough to know what is inside, and the exact version you used has to be named so that someone else can get it again. Each condition can fail separately, so a dataset can be public and still be hard to use honestly.

What makes a dataset open in practice

Four tests help. Can you obtain the data without a private arrangement? Does its licence allow the use you have in mind, and what does it ask in return? Is it documented, so that someone has said what it contains, who made it and what it leaves out? Can you pin the exact version you used? Gebru and colleagues’ “Datasheets for Datasets” (2018) proposes that every dataset come with a short datasheet covering its motivation, composition, collection process and recommended uses.

The last test is easy to skip. A file can be downloadable, but if nobody recorded which revision was used, a result built on it cannot be rebuilt later, even by its own authors.

What Flyingkiwi was trained on

Flyingkiwi’s English and Turkish models were both fine-tuned from Qwen3-0.6B. The table lists every source used, with its licence and the job it did. All of the sources are public, their revisions are pinned, and no private conversations or user data were used.

Three rows need a sentence each. The Aya dataset supplied prompts only, not its answers. The teacher’s answers are not a dataset in the usual sense: an open-weight model wrote them under a written set of rules, and those rules are part of the documentation. The frequency counts from the web and code corpora were used only to choose which tokens to keep in the vocabulary, not to teach the model text.

Reproducibility and auditing

A pinned revision lets someone rebuild the same prompt set. The English fine-tuning used 15,834 training and 344 validation conversations. The Turkish fine-tuning used 20,778 training and 388 validation conversations. Those counts mean little unless the sources are pinned, because a result you cannot rebuild is, in the end, only a claim. Processing needs recording as well as sources. In the concise rewrite, the teacher rewrote 22,067 English answers, with 43% of the characters kept, and 30,036 Turkish answers, with 33% kept. A rewrite that dropped a code block, added a disclaimer, grew longer or switched language kept the original answer. A list of sources alone would not tell a reader that.

Known training data also makes an audit possible. You can search the training set for test questions and see whether any leaked in. Flyingkiwi’s frozen test sets were never uploaded or trained on. The base model’s pretraining data is not public, though, so exposure there is unknown. That is a real limit: the audit covers the project’s own data, not the foundation beneath it. The guide to evaluating a chatbot honestly describes the other habits that keep test sets clean.

Licences need checking one source at a time

Apache-2.0 and MIT both allow reuse on conditions, such as keeping the licence notice. Share-alike licences work differently. The earlier research corpus, labelled v2, used databricks-dolly-15k, which is licensed under CC BY-SA 3.0. That material is not part of v3 or v3.1. Share-alike material is not automatically relicensed under Apache-2.0, so using it would need a separate decision, and that is one reason it was left out. The licences in the table are not all the same. Most rows are Apache-2.0, the GSM8K row is MIT, and the frequency-count sources are ODC-BY or Apache-2.0. Each one carries its own conditions, so each needs its own check rather than a single answer for the whole project.

The practical rule is simple to state and tedious to apply: read the licence of each source, not just the licence of the project that uses it.

Languages depend on the data that exists

Open multilingual data decides which languages a small model can serve. Turkish has less public conversation data than English. The Turkish model therefore drew on prompts from Aya, a community-built multilingual dataset of human-written prompts and answers, and on OASST2 prompts translated into Turkish. Flyingkiwi used only the Aya prompts. The Turkish training set lists 7,761 human-written prompts from OASST2, Aya and translated OASST2, against 7,455 teacher-written requests, so the teacher-written share is close to the human-written one.

Even with that data, the Turkish model scored 37 out of 100 on its frozen set, against 12 for the unchanged base model. That is a large gain and still a weak result, as the article on why Turkish is hard for small language models explains. Open data is necessary for a language like this, but it is not sufficient. The data still has to be cleaned and used with care. Grading used AI graders with spot checks, and there was no independent native-speaker panel, so the results still need checking by people who read Turkish natively.

Open weights are not an open pipeline

Flyingkiwi publishes its open weights under Apache-2.0 and documents its fine-tuning data, so anyone can run the models and inspect how they were trained. The base model’s pretraining data is not public, though. That makes Flyingkiwi’s release open weights, which is a different thing from a fully open training pipeline.

The Open Source AI Definition 1.0, published by the Open Source Initiative in 2024, asks for detailed information about the training data, enough for a skilled person to build a substantially equivalent system. Flyingkiwi documents its own fine-tuning data, but it cannot describe pretraining data it did not collect. On that point the gap is inherited from the base model, and this project cannot close it.

Why the stakes are larger than one small model

A small model makes the problem easy to see. Larger systems face the same question with higher stakes. As systems become more general, what they learn is shaped by the data they see, and by who is able to inspect that data. If the data is closed, the people a system affects often cannot check what it was taught, or test whether it works for their language, their subject or their community. Even a well-documented dataset reflects the choices of the people who collected it, and documentation is what lets others see those choices.

Open, documented data does not guarantee a fair or accurate system. It does make claims checkable, which is the precondition for either. As a reader, the habit is the same as for any AI answer: ask what the system learned from, and whether you could find out for yourself.