The Hallucination Problem: What It Means When AI Lies With Confidence

She went back and read the original article. Whatever she'd asked ChatGPT to summarize or analyze, it hadn't just gotten wrong—it had created something that never existed. Made up citations. Invented facts. Presented them as real.

This is what we call hallucination in AI. Clinical term for something that feels disturbing when you experience it.

What bothers me isn't that AI makes mistakes. Everything makes mistakes. Humans, search engines, books with typos. What bothers me is the structure of the lie. The confidence. The veneer of authority.

She discovered this pattern repeating. She'd ask for information. The model would hallucinate. She'd verify and find fabrication. So she stopped using AI for knowledge work. She went back to what she calls "the tried and tested market." Google. She knows Google has flaws. But she knows how Google fails. She understands the failure modes.

"I've been using Google for decades," she said. "I know the credibility that it provides."

There's something almost defiant about that. In a world racing toward the newest technology, she chose to keep doing what worked. Not because it was perfect. Because it was honest about its imperfections.

I've been thinking about the gap between capability and trustworthiness. A model can be incredibly sophisticated, trained on billions of tokens, able to write poetry and debug code and answer questions across a vast domain. And it can still, with no indication that anything is wrong, make things up.

One of the smartest people working on large language models admitted recently that he doesn't know why hallucinations still happen. He doesn't understand the mechanism. He can't fix what he doesn't understand. Worth sitting with for a moment. We've built these incredibly powerful systems that we can't fully explain, that do things we didn't anticipate, and that occasionally break their own rules in ways we don't have the conceptual framework to fix.

There's something humbling about that. Also something terrifying.

The industry has started responding. Most of the major platforms now have a feature called "deep research" where the model shows its work. It tells you what links it's going to check. It shows you the information it's pulling from. It gives you a window into the process. Some models do this better than others. Some are more convincing than others, regardless of accuracy.

I appreciated how she framed this: "It's just, it just gives more credibility as to what sources is it referring for the information." Not perfect. Not guaranteed. Just slightly more trustworthy because you can see the reasoning.

Here's what I think about when I think about hallucination. It's not a bug. It's not an error that will eventually be fixed with enough scale or enough time. It might be fundamental to how these systems work. They're pattern-matching machines trained to predict the next word. Sometimes the pattern that makes the most sense statistically doesn't align with what's actually true. Sometimes the model is so good at generating plausible text that it generates plausible fiction.

This has real consequences. Someone acting on hallucinated medical information. An executive making decisions based on fake market research. A student citing sources that don't exist.

The problem is worse because the people using these tools often don't know to be skeptical. They see authority. They see detail. They see sourcing. They assume it means accuracy. And sometimes it does. Sometimes it doesn't. The only way to know is to check. Which means the tool only works if you do the work it was supposed to save you from.

Grim irony.

But I want to sit with something she said that I think is important. When she stopped trusting AI for knowledge work, she didn't stop using AI. She just redirected it. She uses it for refinement. For brainstorming. For writing that needs editing. For cases where the output isn't used as ground truth, but as input to her own thinking.

"I've just started using ChatGPT plus, and it's a better model. But before that, when I was on the free version, it's the lower model, and I had to give so much context for everything that I wanted from them."

She upgraded because the better model required less context from her. She could ask a question and get something useful. With the lower model, she had to do so much scaffolding that by the time she was done, she might as well have just done the work herself.

This is where I think the practical wisdom lives. Not "Never use AI." Not "AI is the future, trust it completely." But "Use AI where it genuinely reduces your cognitive load. Don't use it where it just moves the labor around."

The hallucination problem forces a conversation about what we're actually asking these tools to do. We want them to be creative. We want them to synthesize information. We want them to take risks and make connections. Those capabilities are the same ones that lead to hallucination.

You can't have a system that's creative and also perfectly factual. Creativity lives in the space between what's known and what could be imagined. Sometimes that imagination accidentally invents facts. Sometimes it invents beautiful things. The system doesn't know the difference.

So we need humans in that loop. Humans who know to question. Who know to verify. Who can feel when something is off, even if they can't immediately articulate why.

This might be the most important skill of the AI age. Not knowing how to prompt. Not knowing which tool is best. But knowing when to trust a tool and when to trust yourself.

I think about designers in rooms making decisions about features. Designers relying on AI summaries of research they didn't see. Designers accepting outputs without questioning them. And I think about all the small choices that ripple outward. The button that suggests something untrue. The help text that sends someone in the wrong direction. The prediction that assumes something that wasn't true.

We have a responsibility to know what we're building on. To verify. To question. To sometimes just admit that we don't have enough information to make a decision, so we're going to do the work ourselves instead of delegating it to a system we don't fully understand.

That's not resistance. That's design thinking. It's the same rigor that's always been part of this work. It's just more urgent now, because the stakes are higher and the systems are more persuasive.