Ask an AI for academic sources on a niche topic and you may get three neat citations: author, title, year and journal.
Look them up and you might find the author is real, the journal is real, and the paper was never written.
People call this a hallucination. That word makes it sound like a rare bug. It is not. It is the machine doing exactly what it was built to do.
Why it happens
The model predicts what text should come next. When you ask for a citation, the pattern says an author, a year, a title and a journal should follow.
So it writes them. The result looks right, reads well and is completely made up.
There is no step where the model stops, opens a database and checks whether the paper exists. Predicting the next word is the only thing happening.
It has no feeling of doubt
When you do not know something, you feel it. You hesitate. You say “let me check”.
The model has no such feeling. A true fact and an invented one come out of the same process, in the same confident tone.
So confidence tells you nothing. It is just how the tool always sounds.
How this shows up in design work
These mistakes are rarely obvious. That is what makes them risky.
- Research synthesis. You paste twelve interview transcripts and ask for the main themes. You get five. Four are real. The fifth is a theme that often appears in user interviews in general, but nobody in your interviews said it.
- Competitor reviews. You ask how a competitor handles onboarding. You get a clean, believable flow that is really a mix of what most apps do.
- Accessibility. You ask if a layout meets the guidelines. It says yes and quotes a guideline number. The guideline exists, but it says something slightly different.
An obvious mistake costs you nothing, because you spot it. A polished mistake can end up in front of a client.
Three habits that protect you
1. Make it work from your material. Paste the real notes or transcripts and say: “Use only the text above.” This shrinks the room it has to invent.
2. Ask for receipts. Ask for a direct quote or a ticket number next to every claim. For example: “Drop-off at checkout: tickets #1024 and #1089.” Made-up claims have nowhere to hide when you can check them in seconds.
3. Check based on what is at stake. Brainstorming ideas? No need to check. Anything that shapes a roadmap, a client presentation or a legal decision? Check it yourself, every time.
The one line to remember
The model does not know when it does not know. Confidence tells you nothing about correctness. The checking has to come from outside the model, which usually means from you.
What the model can work from depends on what you give it. That is next: what AI can actually see.