A model states, with complete confidence, that a paper exists. Correct-looking authors, plausible title, sensible year. The paper does not exist.
This is usually called hallucination, and it is not a defect that a future patch removes. It follows directly from what the system does.
Where it comes from
A language model generates the most plausible continuation of the text so far. That is the whole objective.
Now consider: what is the most plausible continuation of "the key reference for this is"? It is something that looks like a citation — a name shaped like an author's name, a title shaped like a paper's title, a year in a sensible range.
The model is not consulting a database and misreading it. It is producing text with the right shape. When the shape and the truth coincide, we call it correct. When they do not, we call it a hallucination. The process is identical in both cases, which is exactly why the output gives you no clue which one you are looking at.
When it is most likely
You can predict the risk rather than checking blindly. Fabrication rises when:
- The topic is thin in training data — niche subjects, obscure people, small organisations
- You asked for recall rather than supplying the material
- Specificity is demanded — exact figures, dates, quotations, references
- The question presupposes something false. Ask "why did X do Y?" when X never did Y, and the likely continuation is an explanation, not a correction
- The subject moves quickly and training data has a cutoff
That fourth one is worth testing yourself. Ask about a plausible-sounding thing that does not exist and watch what happens.
What does not fix it
"Do not hallucinate." An instruction cannot supply knowledge. The model has no internal flag marking which of its outputs are grounded.
Low temperature. Temperature controls variety. At zero you get the same answer every time — including the same wrong answer, now more consistently.
Asking it to check its own work. The check is generated by the same process as the original. Agreement measures consistency, not truth.
Asking twice. Same problem. Two plausible continuations, not two independent sources.
What does help
Supply the source. This is by far the largest single improvement. Paste the document. The task changes from recall to reading, and reading is something these systems do very well.
Retrieval. The systematic version of the above: fetch relevant passages automatically and instruct the model to answer only from them. This is what retrieval-augmented generation is for, and why grounded systems are more trustworthy than bare chat.
Demand citations you can check — and then check them. A quoted passage you can locate in the source is verifiable; a reference that merely looks right is not.
Give uncertainty somewhere to go. Explicitly permitting "not stated in the provided text" gives the honest answer a legal form. Without that permission, the most plausible completion is a confident invention.
The right mental model
Treat output as a well-informed draft from someone who never says "I'm not sure". Frequently right, occasionally confidently wrong, and identical in tone either way. That framing gets you the value without the trap — and it is why every stage in this platform pairs a capability with the way it fails.