Open Data for AGI

What the book covers

Discussions of artificial general intelligence usually centre on algorithms and compute. This book is about the ingredient that decides what a system can learn in the first place: the data. It documents the evidence of the past several years and tries to frame the right questions for data policy.

• Descriptions of the major open datasets.
• An analysis of the legal and economic forces that shape data policy.
• A set of recommendations for action.

Where the evidence is contested, the book notes the disagreement rather than resolving it by fiat. It does not claim to have all the answers.

Who it is for

Researchers, engineers, policymakers and citizens who suspect that the future of AGI will be shaped not only by algorithms and compute, but by the data that feeds them. It is written from the perspective of an engineer and designer who treats technical progress and social responsibility as inseparable. At 37 pages, it is a short read.

The argument in two quotations

Open datasets are not a concession to idealism. They are a practical requirement for building systems that are robust, fair, and trustworthy.

AGI is not inevitable. Its form, its distribution, and its consequences are still being negotiated. Data policy is one of the most powerful levers available to shape that negotiation.

Both are from the book’s own description of its aims.

Why it sits next to Flyingkiwi

Flyingkiwi is the argument at a very small scale. Its fine-tuning prompts come from public datasets at pinned revisions, OpenAssistant OASST2, CohereLabs Aya, GSM8K and TriviaQA, under licences that allow reuse; its frozen test sets were kept out of training; and its weights are published under Apache-2.0. That openness is what lets the technical report say exactly what the models learned from and how they were checked.

It also shows the limit. The base model’s own pretraining data is not public, so even this project cannot fully trace what its models saw. The free article Why open datasets matter for AI, from small models to AGI walks through both sides with Flyingkiwi’s sources as the example; the book takes the same question to the scale of AGI.

Details

The views in the book are the author’s own and do not represent any institution or organisation.