What does AGI mean? Definitions, tests and what is still missing

Artificial general intelligence (AGI) is a phrase that appears in research papers, company announcements and news stories alike, yet its meaning is less settled than its frequency suggests. Roughly, it describes a system that can handle a wide range of intellectual tasks, not only the one it was built for. The hard part is turning “wide range” into a test, because different tests reward different things. The definition you accept decides what gets measured, and that decides who can claim progress.

No single agreed definition

There is no single agreed definition of AGI. Four families of definition come up again and again, and each one counts a different kind of evidence. Looking at what each would count as proof, and where each one breaks, is more useful than memorising a slogan. The four are imitation, useful work, graded levels and skill-learning efficiency.

The choice matters in practice. A system that performs well on a school-style exam may still fail a puzzle it has never seen, and a system that sounds convincing in conversation may not complete useful work reliably. Each definition rewards a different strength, and each one leaves a different weakness hidden.

Imitation and useful work

The oldest route comes from Alan Turing. In “Computing Machinery and Intelligence” (1950), published in the journal Mind, he replaced the question “Can machines think?” with an imitation game: a judge exchanges typed messages with a person and a machine, and tries to tell which is which. The evidence is conversational. If the judge cannot reliably tell the two apart, the machine has passed for a person in that exchange. The weakness is that imitation can be gamed. A system can be tuned to sound human without the understanding the test was meant to reveal, and a judge can be swayed by style alone.

The second route asks what a system should do for people. The OpenAI Charter, the company’s public statement of its mission, says that by AGI it means “highly autonomous systems that outperform humans at most economically valuable work”. This definition is practical, because you can ask whether a system does a job as well as a person does. Its weakness is that economically valuable work changes as jobs appear and disappear, and the wording says nothing about how the work is done. A system could meet it through a narrow set of tasks and still be fragile outside them.

Levels and skill-learning efficiency

Morris and colleagues at Google DeepMind, in “Levels of AGI” (2023), grade systems on two separate axes. Performance runs from “Emerging” up to “Superhuman”. Generality runs from narrow to general. The paper also treats autonomy as a separate question rather than folding it into the levels. The evidence is a graded scale that can be checked against agreed tasks. Its weakness is the phrase “agreed tasks”: a level is only as meaningful as the task list behind it, and someone has to choose that list and keep it current.

François Chollet’s “On the Measure of Intelligence” (2019) takes a different angle. It argues that intelligence is better measured by how efficiently a system acquires new skills from little experience. It also introduces ARC, a benchmark of novel grid puzzles that a system is not meant to have seen before. The evidence is performance on tasks designed to be unfamiliar. The weakness is that ARC measures one kind of novelty, built from simple coloured grids. Doing well on it does not, by itself, show competence in language, science or everyday planning.

Why benchmarks age

Any test loses value once it becomes a target. When its questions are posted online, they can end up in later training data, and scores can rise without the underlying ability rising. Researchers call this contamination. You can only check for it if you know what a model was trained on, and many models do not publish that in full.

Flyingkiwi’s frozen test sets were never uploaded or trained on, which rules out one route to contamination. It does not rule out all of them. The base model, Qwen3-0.6B, was pretrained on data that is not public, so even this project cannot say whether test-like text appeared there. The open data article explains why that gap matters.

Where a small model sits

Flyingkiwi’s English model scored 63 out of 100 on its frozen set of project-written cases, the same as the unchanged Qwen3-0.6B it was built from. The grading was done by AI graders against reference answers, so treat that number as an estimate rather than a human verdict. On TruthfulQA MC1, a set of 817 questions where common misconceptions are the tempting wrong answers, it scored 25.1%, against 25.8% for the base model. The gap is inside the test’s sampling error of about 1.5 points, so the honest reading is that the two are roughly level.

The model cannot browse the web, use tools, see your files, remember earlier conversations or know today’s date. Against any of the definitions above, it is nowhere near AGI. It is a narrow tool, useful for some tasks and unreliable for others, and it is best judged as one. The article on what small models can do covers its strengths and failures in more detail.

What is still missing

The four definitions point to gaps rather than dates, so it is more useful to name the gaps than to predict when they will close. The first is reliable facts. A model trained on text can state a wrong thing as fluently as a right one, which is the basic problem behind invented details. A reader meets this whenever a fluent answer is wrong about a name, a date or a figure. The second is knowing what you do not know. Flyingkiwi’s English model said it could not know the answer in 18 of 20 cases where that was the right response. Yet 6 of its 100 replies still contained invented facts or claims.

The third gap is learning a genuinely new task from a few examples, which is the kind of skill Chollet’s measure cares about. This project has no measurement of it, so no number appears here. The fourth is long, multi-step plans, where an early mistake carries into later steps. The project has not measured plans of that length either. The fifth is memory across time. The model does not remember earlier conversations at all, so every new chat starts from nothing.

Why the debate matters to you

Definitions are not only an academic matter. They can shape what regulators ask for, where investment goes and which products get built. A claim that a system is “general” usually depends on the definition behind it, which is one reason to ask what was measured and how. The numbers in this article each measure something narrower than AGI. The 63 out of 100 result scores answers to 100 project-written cases, and the TruthfulQA result scores multiple-choice answers to questions that people often get wrong. Neither is a general intelligence score, and neither should be quoted as one.

You do not need to settle the definition to judge a claim. Ask which test was used, who graded it, and whether its questions were public before the system was tested. The same habits that help you check a chatbot’s answer, covered in how to check an AI answer without starting over, apply to an AGI announcement too.