Back to blog
agents-ia
November 21, 2026
7 min read

Chatbot hallucinations: can an SME actually trust one?

Chatbots sometimes make up facts with total confidence. Why it happens, what it costs an SME, and the safeguards that make a chatbot genuinely reliable.

Vincent

Vincent Roye

Engineer, founder of AI First

Why chatbots make up answers, what that risks for a business, and the safeguards that make them reliable: context, visible sources, human review.
Contents
  1. Why chatbots hallucinate (and why it won't go away)
  2. What hallucinations really cost an SME
  3. How to reduce chatbot hallucinations in a business
  4. Reliable or not: my verdict for SMEs
  5. FAQ

A chatbot hallucination is a wrong answer delivered with exactly the same confidence as a right one. So no, a chatbot left to its own devices can't be trusted to answer your customers or your staff. It becomes reliable only in a specific setup: connected to your own documents, with visible sources, and with a human reviewing anything that commits the business.

I see this problem up close. The experts I lead build AI assistants for SMEs, and the business owner's first question is always the same: "what if it just makes things up?" It's a fair question, and the answer comes down to a few simple mechanisms.

  • ⚠️ A built-in flaw: a chatbot produces what sounds plausible, not what's true, even on simple questions.
  • 📊 A measured cause: OpenAI shows that evaluations reward guessing over admitting ignorance.
  • 🔑 Context first: connecting the chatbot to your own documents cuts off invention at the source.
  • ✅ A clear verdict: reliable with sources and human review, dangerous when answering customers unsupervised.

Why chatbots hallucinate (and why it won't go away)

An AI hallucination is an answer that is unfounded, made up, or inconsistent with the material the model should be relying on, presented as established fact. It isn't a malfunction. It follows from how a large language model (LLM) is built: to produce the most probable continuation of a text.

What does a hallucination actually look like?

The textbook case dates from February 2023. According to Clubic, Google unveiled Bard with a demo in which the chatbot claimed the James Webb telescope had taken the very first pictures of an exoplanet. Those images actually date back to 2004, captured by the European Southern Observatory's Very Large Telescope. Nineteen years off, stated with complete assurance.

Even the term is disputed. In a Know Time video, researcher Nizar Habash points out that the field still debates "hallucination" versus "confabulation", and he brings up Harry Frankfurt's book On Bullshit: a system optimised to produce an answer doesn't care whether that answer is true. I find the comparison more useful than the word itself, because it describes what you're actually dealing with: a model that wants to answer, not a model that wants to be right.

Why does a chatbot guess instead of saying "I don't know"?

Louis-François Bouchard, of the What's AI channel, puts it this way: these models are trained to generate the most likely continuation rather than to say "I don't know". He shows ChatGPT confidently summarising a New York Times article that doesn't exist, based on a fake URL.

In 2025, OpenAI offered a quantifiable explanation for this behaviour. In a study published on 5 September 2025, the company explains that standard evaluations score accuracy alone. As with a multiple-choice test, a random guess might earn a point, while leaving the answer blank earns nothing. So models learn to gamble.

"Language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty."

OpenAI, 5 September 2025

OpenAI illustrates the point with an almost comical test. Asked about the PhD thesis of Adam Tauman Kalai, one of the study's authors, a popular chatbot gave three different titles, all wrong, then three birth dates, all wrong too. Even GPT-5, the company notes, isn't immune. Waiting for the next model to fix the problem is therefore a bad plan.

What hallucinations really cost an SME

For a business, a hallucination becomes costly when the wrong answer commits someone: a customer, an employee, a contract. An error in a draft that a human reviews costs almost nothing. The same error in an automatically sent reply can cost you a dispute.

When does an error become a real risk?

The Agentic AI Institute video gives the most telling example for an SME. An employee asks an assistant how many days of leave they have left. With no access to the company's HR policy, the model makes something up, say, sixty-odd days. The employee plans their holiday around that figure, and the company now has a liability problem, since the information came from a tool it provided.

This isn't just a flaw in the model. It's a flaw in how the project was designed. A model without context fills in the gaps, and a project that doesn't give it the right documents is implicitly asking it to fill in the gaps.

Why are confident chatbots the most dangerous?

According to Clubic, a model doesn't always correctly assess how reliable its own answers are. It may hesitate over accurate information and state an error without blinking. Tone tells you nothing about accuracy.

This is where I push back against a common reflex. Many business owners judge a chatbot by how smoothly it performs in a demo. I judge an AI assistant by what it does when it doesn't know: a useful assistant beats an impressive demo, and a useful assistant knows how to say it's missing a piece of information. On the ongoing cost of errors, see also Fausses alertes IA en entreprise : le coût caché du bruit.

How to reduce chatbot hallucinations in a business

You reduce hallucinations by combining three levers: give the chatbot the right context, require it to admit uncertainty, and put human checks wherever an error is expensive. No single lever is enough, and none of them brings the risk down to zero.

How do you give a chatbot the right context?

Back to the leave example. The Agentic AI Institute suggests adding the HR policy to the system through a connector to the company's folders or SharePoint, then deploying the bot in Slack. The answer changes in kind: the model cites the rule (12 days to start, then 3 weeks once full-time, in the video's example) and, crucially, asks for the employee's length of service and the days already taken before doing the calculation.

That's the behaviour I look for in a project: an assistant that asks for the missing data instead of inventing it. It's exactly what I mean when I say AI delivers value when it's connected to the company's real tools: documents, CRM, database, back office. In an isolated chat window, it improvises.

Is a good prompt enough to prevent errors?

A prompt helps, but it guarantees nothing. TechRadar reports on a prompt that's doing the rounds: ask ChatGPT to act as a "hostile AI auditor" that treats unsupported details as false by default and flags every uncertain claim. The chatbot becomes more cautious and more willing to acknowledge uncertainty.

I use this kind of instruction for my own research, and it does the job well for personal use. But I don't recommend it as a safeguard for customer service, for a reason I've covered in Dérives IA en entreprise : un prompt n'est pas un garde-fou. An instruction can be bypassed. A check built into the process can't.

Which fix for which cause?

The table maps each cause of hallucination cited by the sources to a practical fix for an SME.

Cause What it produces Fix for an SME Implementation cost
Missing context Made-up figures (leave, pricing) Connect the chatbot to the company's documents Medium, an integration project
Guessing instead of admitting Fake source, fake quote Require sources and admission of uncertainty Low, a few lines of instructions
Gaps in the data Wrong historical facts Restrict it to topics your documents cover Low to medium
Uncalibrated confidence Errors stated without hesitation Human review before anything goes out Time, focused on what commits the business

SOURCE: OpenAI, Clubic, cited transcripts · UPDATED 11/2026

Context treats the cause, instructions mask it, and human review catches the rest.

Reliable or not: my verdict for SMEs

A chatbot is reliable in an SME when its scope is limited, its sources are visible and a human signs off on anything that commits the business. It isn't reliable when it's left to answer customers freely about prices, lead times or rules.

Should you avoid chatbots altogether, then?

No, and that's the other mistake. Companies that turn their backs on AI are stuck with costs that are too high and processes that are too slow. Chatbots remain excellent for summarising, rewording, sorting emails or preparing a draft that someone reviews: on those tasks, an occasional error gets caught on reading.

Regulation is heading the same way. European institutions, including the European Commission, already govern the use of AI through transparency and human oversight, two principles this safeguard puts into practice ahead of time.

How do you get started without taking risks?

I'd start with a single, measurable use case where the assistant reads your documents and reports back with its sources. You test the workflow before letting it act on its own, as I explain in Agent IA : testez votre workflow avant qu'il ne casse tout. The three sources I cross-checked all agree: hallucinations don't go away, they get managed.

If you have a specific idea and want to see whether it holds up, you can describe your idea to AI First: an AI expert builds it, a human checks everything, no commitment.

So here's my verdict: trust a chatbot that's connected to your data and reviewed by a human, never one running unchecked in front of your customers.

FAQ

What is a chatbot hallucination?

A chatbot hallucination is a wrong, made-up or unfounded answer presented as established fact. It can take the form of a study, a quote or a figure that doesn't exist. The model isn't lying on purpose: it completes the most plausible answer, even when it lacks reliable information.

Why does ChatGPT make up answers?

According to OpenAI's study of 5 September 2025, standard training and evaluation reward guessing more than admitting uncertainty, so the model learns to guess. On top of that, it's designed to produce the most probable continuation of a text, which favours fluent answers whether they're true or not.

Can hallucinations be eliminated completely?

No. OpenAI acknowledges that hallucinations remain a core problem for large language models, GPT-5 included. You can reduce them significantly by giving the chatbot your documents, requiring sources and having sensitive answers reviewed, but you can never bring them down to zero.

How can I check that a chatbot isn't wrong?

Always ask it for the source of every figure or quote, then open that source. For business use, choose an assistant that answers only from your documents and cites the passage it used. Keep a human review step for anything that commits a customer, a contract or an employee.

Does an "anti-hallucination" prompt work?

It helps, but it isn't enough. An instruction like "act as a hostile auditor and flag uncertain claims", reported by TechRadar, makes ChatGPT more cautious. But instructions can be bypassed: the real safeguard is still the context you give the model and the human check built into the process.

Vidéos YouTube

Articles & ressources

Read next