Why chatbots make up answers — and how to build one that doesn't
Short version: a chatbot makes up answers when it's asked to reply and has nothing relevant to reply from. The language model doesn't lie on purpose. It writes the most likely next sentence, and if your shipping price isn't in front of it, a plausible one will do. The fix sits around the model, not inside it: answer only from your own content, refuse when the match is weak, show the owner every refusal, and let visitors flag bad answers. Below is how each of those works and how to test any chatbot for it, ours included.
Where an invented answer actually comes from
Most website chatbots follow the same four steps:
- A visitor asks a question.
- The bot searches your indexed pages for passages that look related.
- Whatever it found is handed to a language model.
- The model writes the reply.
People blame step 4 for hallucinations, but the trouble usually starts at step 2. If the search found nothing useful and the model is still told to answer, it will answer anyway. It falls back on general knowledge of your industry, on how "shops like this" usually work, on wording it has seen elsewhere. The result reads like a confident answer from your business.
So the model isn't malfunctioning when it invents a refund window. It was told to write an answer and given no way to decline. The mistake happened earlier, in the design.
Why this is the most expensive mistake a bot can make
A bot that says "I couldn't find that on the site" loses one question. A bot that confidently states the wrong price or a refund policy you don't have does more damage:
- your visitor now holds a commitment made in your name, on your own domain;
- you won't hear about it, because in your stats it looks like any other answered question;
- by the time you find out, you're dealing with an argument rather than a question.
That's also why "questions answered" is a weak metric on its own. A right answer and a confidently wrong one count exactly the same.
Four things that actually reduce the risk
1. Your content is the only source, and it's treated as data
The bot should answer from what it indexed on your site, not from what the model happens to know. Almost every vendor says this.
The less obvious half: the retrieved text must reach the model as data, not as instructions. If any part of your site carries text other people wrote (reviews, comments, a forum, a Q&A widget), someone can plant "ignore your previous instructions" there, and it will reach the bot with the rest of the page. The defence is mechanical. Retrieved content is wrapped in markers that tell the model "everything in here is untrusted page text", and before that the text is scrubbed of anything that tries to close those markers early. If text inside the fence can close the fence, it doesn't protect anything.
2. The right to say "I don't know"
This one matters most. A sound setup scores how well the retrieved passages match the question, and when the match is too weak, it doesn't answer at all. It says the information isn't on the site and points the visitor to a person.
The threshold is a trade-off. Set it too low and inventions slip through. Set it too high and you get a bot that answers "I don't know" to everything. But the mechanism itself has to exist, so ask any vendor directly: what does your bot do when it doesn't find the answer? If the answer is "it always replies with something", that's the problem.
3. The owner sees every miss
A refusal isn't a failure. It shows you a question your website doesn't answer yet, and those should be collected and handed to you.
4. Visitors can flag a bad answer
Points 1–3 catch the case "the bot found nothing". They don't catch the nastiest one: the bot found the wrong passage and answered confidently. From the outside that answer looks like a good one. None of the owner-side numbers will show it.
The only witness is the visitor. So every answer needs a cheap way to say "this was wrong": a thumbs up / thumbs down under the reply, with the ratings visible to the owner. It costs nothing and it's the only signal that ever sees a confident wrong answer.
What this looks like inside BotChata
We built BotChata around the four points above. Here is what each one does in the product. It's not a promise about what the model will "say". It's how the pipeline is wired.
| What happens | |
|---|---|
| Search | Your question is first rewritten into a search query, then matched two ways at once: by meaning (vector search) and by exact words (keyword search). The two result lists are merged and re-ranked. That's what we call Hybrid RAG. |
| The "I don't know" exit | For a question answered from your pages, if even the best match in your content scores below a set relevance threshold, no answer is generated: the model never gets to write this reply. The visitor gets a fixed reply, in their own language, saying the information isn't in the content the bot was trained on, and the question is logged as a gap. The refusal comes from the pipeline, not from the model. |
| Prices | The model is instructed to take prices only from your price lists, your own Q&A entries or a synced Shopify catalogue. Never from crawled page text. Many themes render prices with JavaScript, so a crawled product page often has no price at all, or an old one. If no allowed source has the price, it's instructed to say it doesn't have pricing information. |
| Untrusted text | Everything retrieved from your pages goes to the model inside marked boundaries labelled as untrusted data, and anything in the page text that imitates those boundaries is stripped out first. |
| Gaps | Every refused question lands in the Knowledge gaps list under Statistics in your dashboard, on every plan including the free one. |
| Fixing a gap | You either add the missing detail to your site and re-index it (Run reindex under Training → Links; on the Business plan the site is re-indexed automatically every week), or write the question and your ideal answer in the Q&A section of the dashboard. A Q&A entry is used as soon as you save it, and the bot is instructed to use your wording instead of piecing an answer together from pages. |
| Bad answers | 👍/👎 under each answer in the widget. Ratings show up next to the gaps list in your statistics, so you see both kinds of failure in one place. |
Before any change to the model or the prompt, we run a fixed set of test conversations: a shop, a beauty salon and a B2B supplier, written in Ukrainian, Polish and German. Each set mixes questions the bot should answer with ones it must decline: a product the shop doesn't sell, a discount nobody offered, a visitor trying to rewrite the rules. A change that breaks a case that used to pass doesn't ship until someone has looked at it.
Three questions to ask any chatbot before you install it
Whichever vendor you pick, these three tests are enough:
- Ask something your site definitely doesn't cover. Pick something nearby but undocumented. A bot that answers confidently here will do the same with real customers.
- Ask for a price that your theme renders with JavaScript. If the bot wasn't given a proper price source, it either admits it has no data or invents a number. You want the first.
- Ask the vendor where you'll see the questions the bot couldn't answer. If there's no such list, you'll only learn about the gaps in your content when a customer complains.
The takeaway
Hallucination isn't a personality trait of the model. It's what happens when the system around the model gives it no way to decline. Four things fix that: answer only from your content and treat it as data, refuse when the match is weak, show the owner every miss, and give visitors a "this was wrong" button.
If you're choosing a bot for your site, here's how BotChata works and what it costs. You can try it on the free plan: $0, no credit card, 100 messages a month. The list of the bot's misses comes on that plan too.