"Always verify AI output" is advice everyone repeats and nobody follows, because verifying everything costs more than doing the task by hand. The useful version is proportionate checking.
Sort claims by consequence
Before you check anything, ask what happens if this particular sentence is wrong.
Check every time — anything that leaves your desk with your name on it, and specifically: numbers, dates, names, quotations, citations, legal or medical or financial specifics, and any claim you are about to present as fact to someone who will act on it.
Spot check — general explanations in a domain you know, structure and outlines, brainstormed options you will evaluate anyway.
No check needed — text where you are the authority and are simply reading it, format conversions you can see are correct, and anything you are going to rewrite regardless.
Match the check to the claim
- A citation → find the actual source. Not the title in a search box; the document. Invented references are the most common serious error.
- A number → recompute it. If the model both produced and checked it, you have one source twice.
- A factual claim → an independent source, not the same model asked again. Asking twice measures consistency, not truth.
- Code → run it, including edge cases.
- A summary → check it against the original for things added, not just things left out. Added detail is the dangerous direction.
When errors are most likely
You can predict risk rather than checking blindly. It rises when:
- The topic is niche — thin coverage in training data
- You asked for recall rather than giving it the source
- The subject changes quickly, and training data has a cutoff
- The question presupposes something false — models tend to answer the question as posed rather than reject the premise
- You are outside your own expertise, so you cannot see errors
That last one deserves attention. Risk is highest exactly when your ability to detect it is lowest. If you would not notice a mistake, that is the case to verify, not the case to trust.
The structural fix
The most reliable improvement is not better checking — it is changing the task. Give the model the documents instead of asking it to remember. Retrieval-based approaches exist for this reason: grounded in supplied text, the job becomes reading and summarising rather than recalling, and the invented-detail failure mode largely goes away.
Verification is the last defence. Supplying sources is the first, and it is much cheaper.