Six guardrails that stopped our support chatbot inventing answers

A support chatbot that makes things up is worse than no chatbot. A missing answer costs you one email. A confidently wrong answer about a refund policy costs you an argument, and the customer has a screenshot.

"Use a better model" is not the fix. Most of the invented answers I have investigated were not the model being creative — they were the model being handed bad input and doing its best with it. The guardrails below are ordered roughly by how much they bought per hour of work.

1. A retrieval floor, and a real "I don't know" path

This one fixed more than the other five combined.

If your pipeline retrieves the top k chunks and passes them to the model, the model is always given context, even when the question has nothing to do with your content. Ask a plumbing company's bot about visa rules and it gets the three least-irrelevant plumbing paragraphs. Then it writes something.

So: score the best match, and if it falls below a threshold, do not call the generator at all.

const MIN_SCORE = 0.35;   // tune on your own data; this is not universal

async function answer(question) {
  const hits = await retrieve(question, { k: 20 });
  const best = hits[0];

  if (!best || best.score < MIN_SCORE) {
    return {
      text: "I don't have that in our content. I can pass this to the team — " +
            "want me to take your email?",
      sources: [],
      unanswered: true          // log it, see #6
    };
  }
  return generate(question, rerank(question, hits).slice(0, 5));
}

Two notes. The threshold is not transferable between embedding models or even between content sets — measure it. And the fallback has to be a real path, not a dead end: an email capture, a WhatsApp handoff, a phone number. A bot that says "I don't know" and stops is just a slower 404.

2. Make citation structural, not polite

Asking the model nicely to "cite your sources" gets you citations most of the time. Making the output format require them gets you citations you can verify.

Number the chunks in the prompt and require the number inline:

Source [1] (Pricing > Starter plan):
...

Source [2] (Refund policy):
...

Answer the question using only the sources above. After every factual
sentence, put the source number in square brackets. If the sources do not
contain the answer, say exactly: NOT_IN_SOURCES

The point is not prettiness in the UI, although showing the source link does visibly increase trust. The point is that you can now check the work automatically. For every answer, take each cited chunk and ask: does this chunk plausibly support this sentence? A cheap second model call, or even keyword overlap, catches a surprising share of fabrications — and an answer with no citations at all is immediately suspicious.

NOT_IN_SOURCES as a literal sentinel is deliberate. A string you can match beats trying to detect hedging in prose.

3. Strip your own knowledge out of the system prompt

Early on our system prompt contained the pricing tiers, the support hours and the refund window. It made the bot look great in testing.

Then the prices changed. The pages were updated, the index was rebuilt — and the bot kept quoting the old numbers with total confidence, because the system prompt outranked the retrieved content. It took days to find, because every other answer was fine.

The rule I now hold to: the system prompt gets behaviour, never facts. Tone, language, format, what to do when unsure, when to escalate. Anything a customer could be told as a fact lives in the content, where there is exactly one copy of it and updating the page updates the bot.

4. Make "no" the easy answer, with examples

Models hedge toward helpfulness. Telling them "say you don't know if you don't know" is weaker than showing them.

Two or three examples in the prompt do more than a paragraph of instruction:

Q: Do you ship to Dubai?
Sources: [1] Shipping covers India and Nepal.
A: Our shipping page covers India and Nepal [1]. Dubai isn't listed, so I
   can't confirm it — I'll check with the team if you leave your email.

Q: Can I get a refund after 60 days?
Sources: [1] Refunds within 30 days of purchase.
A: NOT_IN_SOURCES

The second example is doing real work. It teaches the model that "30 days" does not license a reasoned extrapolation about 60 days. Without it, models are very willing to reason past the evidence — and that reasoning reads exactly as authoritative as a correct answer.

5. Keep the retrieved context small and ordered

More context is not better. Two things degrade as you stuff the prompt:

Dilution. Twenty chunks with one relevant paragraph is worse than four chunks with one relevant paragraph. The model has more plausible-looking material to blend.

Position. Long-context models still attend unevenly — material in the middle of a long context gets used less reliably than material at the start or end. If you retrieve twenty and keep five, put the best match first.

Four to six chunks after re-ranking has been the sweet spot for us. It is also the cheaper option, which makes it an unusually easy decision.

6. Log every unanswered question and read the log weekly

This is the only guardrail that gets better over time, and it is the one most people skip.

Every time the retrieval floor fires, write down the question. Then read the list.

What comes back is three kinds of entry, and all three are useful:

• Questions your content genuinely does not cover. This is a content backlog written by your customers, prioritised by frequency. It is better input than any keyword tool.
• Questions your content does cover but retrieval missed. These are the real bugs. Usually vocabulary: customers say "bill", your page says "invoice". Fix by adding the customer's word to the page, or with a small synonym map at query time.
• Questions that are not your business at all. Noise, and reassuring — it means the floor is doing its job.

A monthly pass over that list moved our answer rate more than any prompt rewrite.

What this adds up to

Nothing here is clever, and that is roughly the point. Five of the six guardrails are about refusing to answer in the right circumstances, and one is about reading the log.

The uncomfortable truth underneath them: an AI support bot's quality is mostly a property of your documentation, not your model. If the answer is not written down anywhere, the honest output is "I don't know" — and the real work is writing the page.

I build ChatNory, an AI chatbot that answers from a business's own content on its website, WhatsApp and Telegram. All six of these exist because the earlier version did not have them. If you have a guardrail that earned its keep, put it in the comments — I will try it.