We worked through that question while building Insifi, our product that lets business users ask plain-English questions of their company databases. Insifi isn't a RAG system, since it answers from structured data rather than documents. The engineering problem underneath is the one every RAG team faces: an assistant that answers from a source has to know when that source doesn't support an answer.

Our biggest learning from that build was about where verification lives. Checking answers against ground truth, including follow-up questions one level deep, became an evaluation pipeline that runs on every change and shaped the product's architecture. We wrote about that pipeline in what a 50% accuracy score taught us about building AI agents. The next step, and the subject of this blog, is putting verification inside the assistant itself, so it can hold an answer back in the moment.

Most teams building a RAG chatbot for support or internal knowledge focus on retrieval and treat usage as proof it works. The harder problem is deciding when the assistant should say "I don't know," what it says instead, and who picks up the question next. Design all three before launch.

The short answer: a RAG assistant is ready for production when it can reliably:

Answer with citations when the retrieved evidence supports the answer
Refuse, deterministically, once the system has verified the answer isn't in its sources
Hand the question to a named person with enough context that the user never repeats themselves

Usage numbers can't tell you whether any of these work. Testing against known answers can.

Why "people are using it" hides a bad RAG assistant

Usage is the first number most teams report, and early usage is usually flattering. Early questions tend to be simple one-step lookups that any reasonable system gets right. Usage tells you people tried the features. It says little about how the assistant handles a harder question, or a second question in the same thread.

What usage shows	What it can hide
People asking questions every day	Most questions are simple, single-step lookups
No complaints	Users rarely know an answer is wrong unless they check it
Fast, complete answers	Every question gets an answer, including ones the sources can't support
Spot checks pass	Follow-up questions go untested against ground truth

Diagram showing three healthy-looking usage signals (daily questions, no complaints, fast answers) converging on a RAG assistant that feels production-ready until a follow-up question is asked.

The industry has the same blind spot. OpenAI's research on hallucination argues that standard training and evaluation "reward guessing over acknowledging uncertainty." Their SimpleQA comparison shows the cost:

Metric	gpt-5-thinking-mini	OpenAI o4-mini
Abstention rate (no answer given)	52%	1%
Accuracy rate	22%	24%
Error rate (wrong answer)	26%	75%

On an accuracy dashboard, the model that almost never abstains wins by two points. In production, it is wrong nearly three times as often.

RAG doesn't stop confident wrong answers on its own

Retrieval narrows what a model draws on. Nothing forces the model to stay inside it:

Legal research tools: a Stanford study found leading legal AI tools, all using some form of RAG, hallucinated on 17% to 33% of benchmark queries, despite "hallucination-free" marketing in the category.
Air Canada's support chatbot: it told a customer he could claim a bereavement fare after travel, contradicting the airline's own policy. A British Columbia tribunal ordered Air Canada to pay about C$812 and rejected its claim that the chatbot was "a separate legal entity."

Neither case needed a smarter model. Each needed a system that knew when to stop.

When should a RAG chatbot say "I don't know"?

Our position: the assistant should try hard to answer. Once the system has verified the answer isn't in its sources, refusing deterministically is the correct behavior. It should never depend on how the model happens to respond on a given run.

"Verified" carries the weight in that sentence. A prompt saying "answer only from the context" is a request, and the model can't reliably check its own grounding. The check has to live in the system. These are the signals we use:

Signal	What it catches	Where it misleads alone
Scope and risk rules	Topics never to answer alone: refunds, contracts, payroll	Only when the list goes stale
Retrieval relevance score	No relevant source at all	High score on the wrong document, like a similarly named product
Source agreement	Answers resting on one passage, or conflicting sources	Topics documented in only one place
Grounding check	Claims that appear in no retrieved passage	Answers faithful to a stale source
Second-model verifier	Unsupported leaps by the first model	Shared blind spots from the same bad context
Conversation-state check	Follow-ups that change the subject, entity or time range	Short threads where context carries cleanly

No single signal is enough, so we layer them: cheap, certain rules first, retrieval checks next, and a grounding check on every answer that survives. The conversation-state check comes from our own evaluation work: follow-up questions are where answers are most likely to drift, so every follow-up needs its own evidence check. We treat this layering as part of hallucination prevention and handling, one of the eight disciplines in our AI 360 Compass.

Flowchart of the checks a RAG chatbot runs before answering: scope and risk rules, retrieval relevance and grounding, with exits to hand off, refuse or give a partial answer.

How to set the refusal threshold for a RAG chatbot

A confidence number picked from intuition gives you an assistant that refuses everything or nothing. Set the threshold with a golden dataset of verified answers (we covered ours in AI evals for AI agents blog), split into three groups:

Should answer: the answer is in the sources, and the assistant must find it
Should refuse: the answer isn't in the sources, or the topic is out of scope
Should hand off: the answer exists, but a person must own it (refunds, exceptions, anything with liability)

Run the set at several threshold values and record wrong answers given and correct answers withheld. Pick the point where the wrong-answer rate fits that topic's risk. A billing question and a "where's the holiday calendar" question shouldn't share a threshold.

Risk tier	Example questions	Default behavior
Low	Office hours, how-to steps, where a document lives	Answer with citations at a lower evidence bar
Medium	Product specs, plan limits, internal process rules	Answer only with grounded, cited evidence; otherwise give a partial answer
High	Refunds, contract terms, pay, compliance	Draft for a person to confirm, or hand off directly

Give the assistant three ways to respond, not two:

Answer: grounded, cited and specific
Partial answer: "Here's what our documentation says about plan limits. I couldn't find anything on limits for legacy plans."
Refuse and hand off: a clear statement of what couldn't be verified, plus a named next step

The partial answer keeps the user moving and makes the gap visible.

Three response types for a RAG assistant: a cited answer, a partial answer that flags the gap, and a refusal with a hand-off to a person.

The cost of refusing too much

Over-refusal fails quietly. People get "I don't know" a few times, go back to messaging a colleague, and the team reads falling usage as low demand. We treat every refusal as a report on the knowledge base:

What the refusal log shows	Likely cause	Fix
Answer exists in the docs, assistant refused	Retrieval or indexing problem	Fix retrieval; leave the threshold alone
Answer isn't documented anywhere	Content gap	Route to the doc owner; refusals become their backlog
Question is out of scope	Scope rules working	Point the refusal somewhere useful

Lowering the threshold to cut refusals swaps a visible problem for an invisible one. Watch refusal rate by topic, re-ask rate, and how often the person on the hand-off finds the answer in existing docs.

The hand-off to a person: what happens when the assistant gives up

A refusal without a hand-off is a dead end. Gartner found 64% of customers would prefer companies didn't use AI for customer service, and their top concern was that reaching a person would get harder. Our blog on why AI agents fail covers handoffs between agents and systems. This section covers one moment: the assistant gives up, and a person takes over.

Who is the person? A named role with an owner, never a shared inbox:

Customer support chatbot with RAG: the tier-1 agent on shift, with escalation rules for billing and policy exceptions
Internal knowledge assistant: the domain owner, such as HR operations for leave policy
Analytics assistant like Insifi: the analyst who owns that dataset

What the user sees: what couldn't be verified, anything partial that was found, and who will reply and when. For example: "I couldn't find a verified answer on refunds for annual plans cancelled mid-term. I've sent your question to our billing team, who usually reply within four business hours." Our blog on UX patterns that make AI agents trustworthy covers the interface side.

What the person sees: a hand-off packet, so nobody asks the user to start over.

Hand-off field	Why it matters
The question and every follow-up	The user never repeats themselves
Retrieved sources with scores	Shows whether the docs had the answer
Refusal reason (no evidence, failed grounding, scope rule, high risk)	Tells the person what kind of problem it is
The assistant's draft, marked unverified	Saves time when the draft is close
User context (account, plan, role)	No lookup needed

Where the flow breaks. The failure points we design against:

The ticket lands in a queue nobody owns after hours
The person sees only the last message and asks the user to start over
The person's answer never flows back, so the assistant refuses the same question next week

Loop diagram of the hand-off to a person after a RAG assistant refuses, ending with the answer added to the docs and golden dataset.

That last one is the most common. Every human answer to a refused question should become new documentation, a new golden-set test case, or both.

Which RAG evals catch what users complain about

Our evals blog covers eval methods in depth. For a RAG assistant, the narrower question is which signals match what users actually complain about:

Signal	Looks good on a dashboard?	Catches user complaints?
Daily users and query volume	Yes	No
Thumbs-up rate	Yes	Rarely; users can't rate what they can't verify
Single-turn LLM-as-judge scores	Yes	Partly; misses follow-up drift
Multi-turn checks against ground truth	Often worse at first	Yes
Should-refuse pass rate	Rarely tracked	Yes; catches confident answers to unanswerable questions
Sampled hand-off review	Not visible	Yes; shows what users needed and didn't get

Dashboard-friendly signals measure activity. The ones that catch complaints measure behavior under pressure: follow-ups, unanswerable questions, and what happens after a refusal.

Questions to ask before you buy a "connect your docs" RAG chatbot

The pitch usually sounds like "upload your documents and your support chatbot is live by Friday." If I were buying, I'd want these answered first:

Question	What a good answer sounds like
How does it search my documents?	A named retrieval strategy that handles tables, versions and similar names
Does every answer cite its source?	Yes, at passage level, and users can open it
What happens when the answer isn't there?	A testable refusal path, plus a hand-off
What happens when a document changes?	Re-indexing, and invalidation of cached answers built on the old version
How do you test follow-up questions?	Multi-turn evals against ground truth on every change
Who sees the refused questions?	A named owner and a review loop

The caching question trips up the most vendors. An assistant that cached last quarter's pricing answer will keep giving it, confidently, after the pricing page changes.

Ship the "I don't know" before you ship the answer

The lesson from building Insifi carries straight over to RAG: verification belongs inside the product as well as in the test suite. An assistant that can't hold back makes every answer look equally trustworthy, and users can't tell which ones to trust. Teams that get RAG for support and knowledge right design three things before launch: the evidence bar for answering, a refusal that keeps the user moving, and the person who takes over.

If you remember one thing: a refusal you designed is a feature, and a wrong answer you didn't catch is a liability with your logo on it.

Test your assistant with questions it should refuse

Before your next release, write 20 questions your assistant has no business answering: out-of-scope topics, questions your docs don't cover, and follow-ups that change the subject. Run them. If it confidently answers more than a couple, you have a refusal problem to fix before users find it.

Aubergine’s Applied AI team builds RAG assistants with refusal, hand-off and evaluation in place from day one. Planning a support or knowledge assistant, or unsure what your live one does with hard questions? Let’s talk.

FAQs

What is a good refusal rate for a RAG support chatbot?

There isn't a universal number. It depends on how complete your documentation is and how risky the topics are. Track refusal rate by topic instead of one overall figure, and watch how it moves after each content update. A rising rate in one topic usually points to a content gap or a retrieval bug in that area.

Can I fix hallucinations by telling the model to say "I don't know" in the prompt?

It helps at the margins, though a prompt instruction is still a request the model can ignore on a given run. Reliable refusal needs checks outside the model: retrieval thresholds, grounding checks and scope rules the system enforces regardless of what the model generated.

Should a chatbot tell users when it's handing them to a human?

Yes. Say what the assistant couldn't verify, who will pick it up and roughly when. Gartner advises that chatbots tell customers they'll connect them to an agent when AI can't solve the issue, and that the agent chat should pick up where the bot left off.

How do I stop a RAG chatbot from giving outdated answers after documents change?

Re-index changed documents on a schedule or on every update, and invalidate cached answers built on the old version. Add the changed topics to your regression evals so stale answers get caught before users see them.

Is a company responsible for what its AI chatbot tells customers?

In the Air Canada case, a Canadian tribunal held the airline responsible for its chatbot's incorrect advice. Rules vary by country and industry, so check with legal counsel. A safe working assumption: your chatbot's answers carry the same weight as your website's.

Does RAG eliminate hallucinations?

No. RAG reduces them by grounding answers in your content, yet the Stanford study of legal research tools still found hallucinations on 17% to 33% of queries. Retrieval quality, grounding checks and refusal design decide how often a wrong answer reaches a user.

Should we add refusal to an assistant that's already live, or rebuild it?

Usually add. Start with scope rules and a should-refuse test set, since both sit on top of an existing system. Grounding checks and a proper hand-off packet may need architectural changes. Building Insifi's evaluation pipeline involved architectural changes too, so plan for them.

Download report

Authors

Harsh Soni

Associate Technical Lead

Harsh is a developer with about a decade (if not more) of expertise in building mobile apps. More of a doer, less of a talker, he likes to keep his mind occupied, and has an eye for understanding systems in their barebones, helping him be his creative best, impacting users' lives for the better. Enjoys building reusable & scalable systems, reading a book in a quiet beautiful place, or learning something new.

View Author