Hallucinating intelligence
AbstractLanguage models are trained to write what sounds right. Most of the time that is also what is true, which is exactly why their mistakes are so hard to catch. We sort the questions an agent faces by the kind of compute that should settle each one, and list what would prove us wrong.
A language model writes the most plausible continuation of what it is given. A wrong answer has the same facade of sound reasoning as a right one. The hallucination worth worrying about may be ours. Text with the surface of intelligence gets read as intelligence.
Fiction is the job
Given a piece of text, a language model predicts the likelihood of every choice of the next word, based on experience reading lots of text. Nothing in that objective refers to the world, so a common false sentence is cheaper for the model than a rare true one. The model has learned how text behaves, and text tracks the world most of the time.
Léon Bottou and Bernhard Schölkopf picture the apparatus:[1]
Imagine a long paper tape with a few initial words of a text. An apparatus scans the paper tape, randomly picks an occurrence of this sequence of words in our infinite collection, finds the word that follows this occurrence, and prints it on the tape right after the previous words.
At any instant, our imagined apparatus is about to generate a story constrained by the narrative demands of what is already printed on the tape. Some words were typed by the user, some result from the past random picks of the language model. Neither truth nor intention matters to the operation of the machine, only narrative necessity.
Often the continuation narrative necessity demands happens to be true, and that is why the trick works. When it is false the model has done the same job. There is no separate mode for making things up that you could switch off. Models decode a letter-shift cipher far better at the shift common on the internet than at a neighbouring one, so accuracy tracks how probable the answer is as text, alongside how hard the problem is.[2] Accuracy on multi-digit multiplication collapses as the numbers grow.[3] As Kambhampati and colleagues put it, "a system that takes constant time to produce the next token cannot possibly be doing principled reasoning on its own."[4]
Fluent language, no executive
A person with fluent language and impaired executive function is a familiar figure in neurology. In Shallice and Burgess's study of frontal lobe patients, people with normal intelligence and intact language were sent on simple errands. They understood the rules. They still broke them and left tasks unfinished.[5] Duncan and colleagues called part of this pattern goal neglect: a person can state the instruction correctly and then fail to act on it.[6]
Today's agents fail the same way. They recite the policy and break it, lose the objective halfway, retry a failed step, and report success on unfinished work. Better models improve the language area. The executive function, the prefrontal cortex, is still missing.
Different questions need different compute
Making one model do everything means deriving the world from scratch, every time. A plausible plan, a valid inference, an action the policy allows and an intervention that will cause what you want are four different things, settled by four different kinds of evidence. A language model can write a convincing answer to all four. That does not make it the authority on any of them.
| Kind of question | Example | Compute that settles it |
|---|---|---|
| Meaning | What does this person want? | A language model's semantic judgement |
| Quantity | What is the total, the deadline, the limit? | Exact calculation |
| Feasibility and choice | Is there a plan that meets every constraint? | Search, planning and solvers |
| Permission | Do the rules allow this action? | Rules derived from the written policy |
| Effect | Will this action cause the outcome we want? | Causal inference, assumptions stated |
| Value | What should we do when the rules run out? | A person who owns the decision |
For the first row a language model is the best general instrument we have. For the rest, when it is right, nothing about how it got there guarantees it. Each kind of compute also proves less than it seems to. A perfect solver run over an incomplete encoding still fails end to end. So the work moves to translation, and to picking the compute each question needs. That choice is the subject of our approach note, Recipes, abstractions and taste.
How to prove us wrong
- Surface changes stop mattering. Without external tools, a frontier model shows no measurable gap between familiar problems and versions with renamed entities, changed numbers and irrelevant clauses, on planning, arithmetic and policy tasks.
- Scale stops separating the methods. On constraint problems, frontier models match exact search on large instances in the same time.
If you have the evidence first, we would like to hear it.
References
-
Léon Bottou and Bernhard Schölkopf, "Borges and AI," arXiv:2310.01425, 2023, §1. ↩
-
R. Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D. Hardy and Thomas L. Griffiths, "Embers of autoregression show how large language models are shaped by the problem they are trained to solve," Proceedings of the National Academy of Sciences 121(41), 2024. ↩
-
Nouha Dziri et al., "Faith and Fate: Limits of Transformers on Compositionality," NeurIPS 2023. ↩
-
Subbarao Kambhampati et al., "LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks," ICML 2024, §1, arXiv:2402.01817. ↩
-
Tim Shallice and Paul W. Burgess, "Deficits in strategy application following frontal lobe damage in man," Brain 114(2), 1991. ↩
-
John Duncan, Hazel Emslie, Phyllis Williams, Roger Johnson and Charles Freer, "Intelligence and the frontal lobe: The organization of goal-directed behavior," Cognitive Psychology 30(3), 1996. ↩