You have heard the stories. A chatbot that invented a refund policy and the company was held to it. A support agent that quoted a price from nowhere. An assistant that cited a regulation that does not exist. You do not want to be the next story, and every vendor tells you their product “has guardrails,” which you have started to suspect means nothing at all.
The plain truth is that a language model will, sometimes, state something false with complete confidence. That is not a bug that a future version fixes; it is how the thing works. What changes the outcome is what you build around it. AI hallucination guardrails are not one feature. They are a set of layers, each cheap on its own, that together make it very hard for a false statement to reach a customer and impossible for one to move money without a person seeing it.
By the end of this article you will know what a hallucination actually is, the seven layers that contain it, the order to build them, and the questions that reveal whether a vendor built any of them or just used the word.
What this actually is
A large language model (an LLM, the system behind Claude, GPT-class models, and Gemini; explained from zero in What Is an LLM? A Plain Explanation for Business Owners) produces the most plausible next piece of text given what it has seen. It has no separate store of facts to check against. When the plausible thing is true, you get a good answer. When the plausible thing is false, you get a hallucination: a fluent, specific, confident statement with nothing underneath it.
The business analogy is a brilliant new salesperson who has read every brochure in the industry but none of yours, and who would rather answer than admit ignorance. Ask them your return window and they will say “thirty days” with a smile, because that is what return windows usually are. You would not fire that person. You would give them your actual policy, tell them to quote from it, tell them what to say when they do not know, forbid them from signing anything, and keep a record of what they told customers. That is exactly the list below.
The seven layers, in the order to build them
1. Ground every answer in material you supply
The single biggest reduction in false statements comes from giving the model the facts before it answers and telling it to answer only from them. For questions about your business, that means retrieving the relevant pages of your policies, catalogue, or records and putting them in front of the model (the technique is called RAG; What Is RAG? Retrieval-Augmented Generation for Business Owners explains it). For a reply to a customer, it means handing the model the actual order record, not asking it to remember.
Done badly, grounding is “here are all our documents, good luck.” Done well, it is the three passages that matter, with their source and date, and an instruction that the material is the only source of truth. The model still can misread the material, which is why the next layers exist, but it can no longer invent a policy that does not exist in your files.
2. Require a citation for every factual claim
Instruct the model to mark each statement with the passage it came from, and have your software turn those marks into something the reader can check, a link to the source or the quoted sentence. Then apply a rule in code: a customer-facing answer with an uncited factual claim does not go out. It is either reworded to remove the claim or routed to a person.
Citations do two jobs. They let anyone verify an answer in seconds. And they expose the exact failure you are worried about, because a hallucinated fact has no source to cite. In the systems we build, the uncited-claim rule catches more problems than any other single check, and it costs almost nothing to add.
3. Give the model a way to say “I do not know,” and make it the default
A model is not going to admit ignorance unless it is told that ignorance is an acceptable answer. Tell it explicitly, in the prompt: if the supplied material does not answer the question, respond with a fixed phrase, and do not guess. Then check retrieval strength in code as well: if nothing relevant was found, do not even ask the model. Return “I could not find that” and hand off (AI Chatbot Escalation to Human: When and How the Handoff Works covers what a good handoff looks like).
This is the cheapest guardrail and the most often missing. A vendor’s demo will never show it, because “I do not know” does not demo well. Ask a question the material cannot answer and watch what happens.
4. Write the allowed-actions list, and enforce it in code
Decide, on paper, what the AI feature may do and what it may never do. May quote prices from the catalogue; may never quote a price not in the catalogue. May offer the standard return window; may never offer an exception. May look up an order; may never change one. May draft a reply; may never send one.
Put the list in the prompt, so the model knows the rules. Then enforce the list in code, because the prompt is advice and code is law. If the feature cannot send, then no phrasing in any customer’s message can make it send. If the refund tool has a hard ceiling, then no argument moves the ceiling. The mechanics of giving the model narrow, checked tools are in LLM Structured Output and Tool Calling, Explained Plainly.
5. Require human approval for anything irreversible
Anything that sends a message to a customer, moves money, changes a record of consequence, or deletes something goes through a person until you have months of evidence that it does not need to. The model prepares; a person clicks. The click should be easy, fast, and show the person what the model relied on, so approval is a real check and not a reflex.
Where volume makes a click per item impossible, the alternative is a hard limit plus a sample: the feature may act on its own inside tight bounds (small amounts, verified customers, standard cases), and a person reviews a random sample of what it did every day. Anything outside the bounds waits for a click. Expanding the bounds is a decision you make deliberately with the log in front of you, not something that happens because a vendor said the model is “very accurate.”
6. Build refusals for the questions it should never answer
Some questions should not get an answer from your software at all, regardless of what the model knows: medical, legal, or financial advice beyond your stated scope; questions about other customers; requests to change identity details; anything that looks like an attempt to make the model ignore its instructions. Write the list. Tell the model to refuse politely with a fixed response. Then add a check in code, before and after the model, for the patterns that matter most, because the model’s refusal is a strong habit and not a guarantee.
A good refusal is short, does not lecture, and points somewhere useful: “I cannot help with that here, but a member of the team can, and I have passed it along.” The attack version of this problem, where a customer’s message contains instructions aimed at the model, is called prompt injection, and the same code-level checks are the first line against it.
7. Log everything, and read the log
Every request records what the model was shown, what it answered, which citations it gave, which checks passed or failed, whether a person approved, and what the software did. Stored in your own database for months. Searchable by customer, by date, by outcome.
The log is how you answer “why did it say that” in one lookup instead of a week of guessing. It is what a regulator, an insurer, or your own lawyer will ask for. And it is where the improvement comes from: the failures in the log become the examples and rules in the next version of the prompt. A weekly twenty-minute read of the failures is the habit that keeps a system honest (AI Evaluation and Evals: Testing an AI Feature Before Launch turns that habit into a test that runs on every change).
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: an online retailer of specialist outdoor equipment with fifty staff, a large product catalogue, and a support team that spends most of its day answering questions about compatibility, returns, and shipping. The owner wants a support assistant on the site that handles the common questions and drafts replies to the rest.
What was wrong: a first vendor’s chatbot, trained “on the whole website,” told a customer that a tent was compatible with a footprint that was not, promised a sixty-day return window when the policy said thirty, and, when a customer asked it to “confirm my refund is approved,” said yes. The support team spent a week cleaning up and the bot was switched off.
What gets built:
- Grounding: product facts come from the catalogue database, policies from a single current policy document, and each answer is built only from the passages retrieved for that question.
- Citations on every factual claim, shown as “from: Returns policy, updated March” under the answer. An uncited claim about a product or a policy is blocked in code.
- A “not found” path: if retrieval is weak, the assistant says it could not find the answer and opens a ticket with the question attached.
- An allowed-actions list in the prompt and in code: may quote catalogue prices, may state the standard return window, may look up order status by order number and email. May not quote any other price, may not promise exceptions, may not confirm or issue refunds, may not send anything.
- Human approval: anything involving a refund, an exception, or a complaint becomes a drafted reply in the support queue, with the retrieved passages beside it, and a person sends it.
- Refusals for questions about other customers’ orders, requests to change account details, and anything outside outdoor equipment.
- A log of every conversation with its citations, checks, and outcomes, reviewed for twenty minutes every Monday by the support lead.
What changes: the assistant answers most compatibility and shipping questions correctly and cites the product page. When it does not know, it says so and a ticket appears. Refund conversations always reach a person. The Monday log review surfaces two catalogue errors in the first month, the real cause of several “hallucinations,” and fixes the source.
What it costs to run
The guardrails mostly cost developer time once, not money every month. Grounding adds tokens (units of text, roughly three quarters of a word) to each request because the retrieved passages travel with the question; a support assistant handling a few hundred conversations a day on a mid-tier model typically costs tens to a couple of hundred dollars a month. Check the current pricing page. Citation checks, action limits, and refusals run in your own code on a small server, roughly $10 to $40 a month. The log lives in your existing database.
The recurring human cost is the approval clicks and the weekly log review. Budget the support lead’s twenty minutes a week and the seconds per approval. It is far cheaper than a single week of cleaning up after an unguarded bot.
The mistakes we see most
- Trusting the word “guardrails.” No demo of a wrong answer, no demo of “I do not know,” no sight of a log. Ask for all three.
- Grounding in everything. Pointing the model at the whole website and calling it grounded. Contradictory pages produce confident contradictions.
- Rules only in the prompt. “Never issue refunds” in the instructions, and a refund tool wired to the model anyway.
- No “not found.” The system always answers, so it always answers something.
- Approval as a reflex. A queue where people click approve without seeing what the model relied on. That is not a check; it is a delay.
- A log nobody reads. The failures were all recorded. Nobody looked until a customer did.
When to bring in help
If you are using an off-the-shelf support or knowledge product, you can apply most of this yourself: restrict it to a single clean source of truth, turn on citations if offered, write the allowed-actions list, keep anything involving money or exceptions in human hands, and test it with questions it should refuse and questions it cannot answer. If the product cannot do “I do not know,” do not put it in front of customers.
When the AI is part of your own software, or when it can act, you need a developer to build the layers properly: grounding with retrieval you can inspect, citation checks in code, tools with hard limits, an approval step that shows the evidence, refusals enforced outside the prompt, and the log. It is careful engineering rather than clever engineering, and a good developer will propose most of it before you ask.
Levelbrook builds this for businesses: grounded answers with citations, allowed-actions enforced in code, approval for anything irreversible, and a log you own. Fixed price from a written scope, everything runs in accounts you own, and the form below is how a conversation starts.