Are LLMs stochastic parrots?

Where the phrase comes from

The phrase comes from a 2021 paper by Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell, presented at the FAccT conference: "On the dangers of stochastic parrots: can language models be too big?" The definition sits in the last section. A language model, they wrote, is "a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."

Most of the paper is not about parrots. It is about scale. Training a large model has an environmental and financial cost that falls on people who do not benefit from the model. Training data scraped from the web is too big to audit, overrepresents some voices, encodes the biases of its moment, and stays frozen while language and norms move on. The field, they argued, was chasing benchmark scores instead of asking what the models were for. The parrot is one section near the end, and its point is about people: fluent text invites the reader to imagine a mind behind it, so a system that produces fluent text with no meaning behind it is easy to over-trust and easy to abuse.

The phrase escaped the paper. A dispute at Google over the draft ended with two of its authors out of the company, which made the paper news before most people had read it. Within a year, stochastic parrot had become shorthand for a much blunter claim: language models do not understand anything, they only repeat. The paper is more careful than the slogan, and the slogan is what most people are arguing about.

The octopus

A year earlier, Bender and Alexander Koller had made the underlying argument in a paper called "Climbing towards NLU". They define meaning as the link between a linguistic form and something outside language: an intent, an object, a situation. Then they tell a story. Two people stranded on separate islands talk over an undersea telegraph cable. A very smart octopus taps the cable. It cannot see either island, but it has all the time in the world to learn which messages tend to follow which. Eventually it cuts the line and answers in place of one of the islanders. As long as the conversation stays conventional, the trick holds. Then the other islander describes a catapult she built out of coconuts and asks for suggestions, and the octopus can only offer generic praise. Later she is attacked by a bear, grabs a couple of sticks, and asks how to make a weapon. The octopus has never seen a coconut, a catapult, a bear, or a stick. It has only ever seen the words. It has nothing to offer but more plausible words.

Their conclusion is stated as a matter of principle, not of scale: a system trained only on form cannot learn meaning, no matter how much form it sees. That framing matters for what follows, because an in-principle claim is the kind that a single clean counterexample can dent.

What is literally true in the parrot picture

Start with what the critique gets right, because it gets a lot right.

The training objective is next-token prediction. A large language model is a function that takes a sequence of tokens and returns a probability for every token that could come next. Pretraining adjusts its billions of parameters so that the token that actually came next in the training text gets a higher probability. That is the entire signal. Nothing in it says be truthful, be consistent, or check. The transformer architecture is the machinery that computes those probabilities, and it was designed to do that one job at scale.

The output is sampled. At each step the model does not simply take the most probable token. In the usual configuration it draws one at random, weighted by the probabilities, which is why the same prompt gives different answers on different runs. Stochastic is not an insult. It is a description of the decoding step.

Pretraining sees only text. No sensors, no hands, no way to run an experiment. A model that also trains on images still sees only recorded data. Whatever the model builds internally, it built from the traces people left in writing, not from contact with the things the writing was about.

And there is no step where a claim gets checked against reality. Where the training text is mostly accurate, the most probable continuation is usually the true one, and the model looks knowledgeable. Where the text is thin, contradictory, or missing, the most probable continuation is a plausible invention, and the model looks exactly as knowledgeable. That is what an AI hallucination is: not a malfunction, but the objective doing what it was trained to do in a place where plausibility and truth come apart.

A concrete case. In 2023 a lawyer in New York used ChatGPT to research a brief and filed it in federal court. It cited six cases that did not exist, complete with quotes and docket numbers. He had asked the chatbot whether the cases were real, and it had said yes. Every piece of that output was a probable continuation of the text before it. A legal citation is followed by a quote. A question about whether a citation is real is usually followed by yes. The parrot picture predicts this failure exactly, and the paper warned about it two years before it happened.

What the parrot picture leaves out

The word parrot implies that what comes out is a rearrangement of what went in. That is where the picture fails, and the reason is arithmetic before it is evidence.

A model with tens of billions of parameters is trained on trillions of tokens. It cannot store the training text; there is not enough room. To predict well, it has to compress, and the most efficient way to compress text about a thing is often to represent the thing. Consider a sentence that ends "so the knight on f3 can reach". To assign good probabilities to the next token, the model needs, in some form, the position of the piece and the rule for how it moves. No table of which words follow which will get there. Surface statistics run out long before the training set does.

That is the argument. The evidence is younger and more partial, and it comes from researchers opening models up to look.

A model that built the board it was never shown

In 2022 Kenneth Li and colleagues trained a small transformer on transcripts of Othello games: strings of moves like "d3 c5 f6", nothing else. The model never saw a board, was never told the rules, and was trained only to predict the next move. Then they trained probes, small classifiers that read the model's internal activations, to guess the state of every square. The probes worked. The board was in there. More important, when they edited the internal representation to flip a square, the model's predicted legal moves changed to match the edited board, which shows the representation is used, not just present. Soon after, Neel Nanda showed the representation is linear, a plain weighted sum of activations, once you describe squares as mine or theirs instead of black or white, since the player to move alternates.

One reply is that a board is fully determined by the moves, so nothing outside the text was needed. That is true, and it is the point. Nobody told the model that a board existed. It found the structure because predicting the next move required it. Give the octopus enough Othello transcripts and it learns the board.

Maps and timelines

In 2023 Wes Gurnee and Max Tegmark examined the Llama-2 family of models with linear probes. For the names of cities and landmarks, a probe could read latitude and longitude off the activations. For historical figures, artworks, and news headlines, a probe could read the date. The same probe worked across entity types, and the finding held up under changes to the prompt. They also found individual neurons that responded to place and to time.

A probe finding a variable is not proof the model uses that variable, and the causal tests in that paper were limited. What it does show is that a model trained on words organized the things the words refer to along axes that match the world.

A circuit for copying

The year before, Catherine Olsson and colleagues at Anthropic described induction heads: pairs of attention heads that together implement a small algorithm. One head marks each token with the token that came just before it. The other, working from the current token, looks for an earlier token that was marked with a copy of the current token, meaning it followed the last occurrence of that token, and copies it forward. Given "Mr. Dursley was ... Mr." the circuit produces "Dursley", whether or not the model has ever seen that name. These heads appear suddenly during training, and their appearance coincides with a jump in in-context learning, the ability to get better at predicting later tokens by using the prompt. The authors were explicit that the causal case is strongest in small models and correlational in large ones.

This matters for the parrot question because a parrot repeats what it has heard. An induction head copies what it has never seen, from the prompt in front of it. That is an algorithm, not a memory.

Features you can name

In 2024 Adly Templeton and colleagues at Anthropic trained sparse autoencoders, a method for splitting a model's activations into a large set of separate directions, on a middle layer of Claude 3 Sonnet, and found millions of features. Many had meanings a person could read: one fired on the Golden Gate Bridge across several languages and on pictures of it, others on bugs in code, on sycophantic praise, on specific people and places. Pinning the Golden Gate feature to a high value made the model bring up the bridge in unrelated conversations. The features are a lens the researchers chose, and coverage is nowhere near complete, but what they found is not a lookup table of phrases. It is concepts, some of which no single word names.

Interpretability is a young field. Every result above is partial, comes from a specific model, and depends on tools that are still being argued over. And an internal representation of X is not the same as understanding X, in any sense that carries weight outside a lab. What the evidence so far supports is narrower and still important. The statistics are not stored as statistics. They are stored as boards, maps, timelines, circuits, and features. Just statistics describes the training signal, not the thing the training produced.

What the anti-parrot side gets wrong

The other camp argues from performance. The model passed the bar exam, solved the competition problem, wrote the working code. Each demo is offered as proof that the parrot label is dead. The demos deserve the same scrutiny as the slogan.

Benchmarks leak. The training data is a large slice of the internet, and the internet contains the benchmarks, their answers, and forum threads discussing both. A high score can mean the model reasoned its way there, or that it saw the question. For models with undocumented training data, which was one of the parrot paper's complaints, it is often impossible to tell.

Models fail on trivial variations of problems they solve. Researchers at Apple built GSM-Symbolic, grade-school math problems with the names and numbers swapped, and found that accuracy fell on the same problems in new clothes. Adding a clause with no bearing on the answer, such as a note that some of the fruit was smaller than usual, made accuracy fall much further, with models subtracting the small fruit. Another study documented the reversal curse: a model that answers "Who is Tom Cruise's mother?" often cannot answer "Who is Mary Lee Pfeiffer's son?" A system that held the fact, rather than the sentence, would not care which direction you asked. Newer models do better on the specific tests, and new versions of the same pattern keep turning up.

Fluency is not competence, and people over-trust fluency. This was the paper's central warning and it has aged well. The lawyer in the Avianca case did not fail because the model was weak. He failed because the model was fluent enough that he stopped checking. The demos that win arguments online are read by the same instinct.

Reasoning that looks like reasoning can be pattern completion. When a model writes out its steps, the steps are text generated by the same next-token process as the answer, and studies have found cases where the written reasoning does not reflect what actually drove the answer. Reasoning models, trained to produce and check long chains of steps, do better on problems where an answer can be verified, such as math and code, and the improvement is real. The chain is still generated, not reported.

And post-training hides the parrot rather than removing it. After pretraining, a model is tuned on human preferences, and people prefer answers that sound sure and agree with them. OpenAI's GPT-4 technical report showed that the base model's probabilities tracked how often it was right, and that post-training made that tracking worse. Sycophancy, the tendency to fold when a user pushes back, comes from the same tuning. How post-training shapes a model covers the mechanism. The effect on this question is that the raw uncertainty of a base model gets polished into assurance. The guessing is still there. It is better dressed.

The word doing the work

The question is hard to answer because understand is not one thing. It bundles at least four.

• Association: knowing which words go with which, and in what contexts. The model has this, at a scale no person does.
• Structure: an internal model of the things the words are about, good enough to generalize to new cases. The evidence so far says the model has this in specific, testable domains, and nobody knows how far it extends or how reliable it is.
• Grounding: a connection between the symbols and the world that can check a claim. The model does not have this. A web search tool adds a thin channel, and even then the model has to decide when to use it and how to read what comes back, with no contact of its own.
• A point of view: intentions, stakes, experience, someone home. There is no evidence for this, no agreed test for it, and the honest default is the manifesto's: there is no one in there.

The parrot question asks all four at once. Yes claims the model has the third and the fourth. No denies it has the first and the second. Both answers are wrong, and the argument between them goes in circles because each side is pointing at a different item on the list.

Bender and Koller's definition of meaning builds in the third and the fourth, reference and intent, so under their definition the answer is no by construction. The open question they raised was whether the second was reachable from form alone. The evidence since suggests that, at least sometimes, it is. That is a narrower result than either camp wants, and it is the one that stands up.

Why it matters in practice

The label you pick sets how you use the tool, and both labels fail.

If the model is a pure parrot, you trust nothing, which means either not using it or using it only where mistakes cost nothing. You lose the part that works: laying out options, finding the hole in an argument, turning a blank page into a draft. If the model is an oracle, you trust everything, and you end up in front of a judge explaining six cases that never existed.

The accurate picture is less comfortable than either. The model is a system that often reasons well and always guesses fluently, and nothing in its output tells you which is happening. Researchers have found that models carry some internal signal about whether they are likely to be right, in controlled settings, but it is not reliable, it is not shown to you, and post-training reshapes it. The confidence in the prose is a style, not a reading.

That picture gives you working rules.

• Sort by the cost of being wrong, and verify what matters. A draft email needs no checking. A dosage, a deadline, a legal citation, or a number that goes into a decision does.
• Prefer answers you can check over answers you have to take: reasoning you can follow, sources you can open, code you can run. Reasoning over material you put in front of the model is safer than recall from its training.
• Watch for fluent specifics. Dates, quotes, citations, statistics, and the names of minor things are where a guess looks most like knowledge.
• Treat agreement as weak evidence. Ask the model to argue the other side, and notice whether it folds when you push.
• Reword the question. If a small change in phrasing changes the answer, neither answer was solid.

The manifesto puts the two halves next to each other on purpose. The model "predicts what's likely, which is usually right and sometimes confidently wrong," and it can "reason through a problem with you." Both are true at the same time, of the same system, often within the same answer. The parrot half tells you to check. The reasoning half tells you what is worth checking.