You have a system, or a proposal for one, that answers questions from your own documents. The demo was good. Then a real question came in, the answer was wrong, and nobody could explain why. Or the answers are right most of the time and you have no idea what “most” means.
Everyone who builds one of these learns the same lesson: the model is rarely the problem. The problem is one of a handful of unglamorous parts between your documents and the model, and each one fails in a predictable way. This article walks through RAG implementation best practices part by part: what each one is, how it goes wrong, and what done-properly looks like. It is written so you can judge whether your vendor did the work, and so a developer can act on it.
If you want the concept first, read What Is RAG? Retrieval-Augmented Generation for Business Owners. This article assumes you know the library picture: a model (the librarian) answering from pages an assistant (retrieval) pulled from your binder (the knowledge base).
What this actually is
A RAG system (retrieval-augmented generation: find the relevant pieces of your material, hand them to the model, have it answer from those pieces) is a pipeline. Documents go in one end, cut into pieces, mapped by meaning, stored. Questions come in the other end, matched to the pieces, and the best pieces and the question go to the model together.
The business analogy is a mail room. Everything about whether the right letter reaches the right desk depends on how the mail is sorted, labeled, and filed, not on how clever the person at the desk is. A brilliant executive with a broken mail room answers the wrong letters. Every practice below is about the mail room.
The eight parts, and how to build each one
1. Chunk along the document’s own structure
Chunking is cutting documents into pieces small enough to search and hand to the model. The lazy way is every 500 words, cut wherever that lands. That splits tables, separates a rule from its exception, and strands a paragraph from the heading that gave it meaning.
Cut along structure instead: headings, sections, list items, table rows kept whole. Prepend each chunk with its breadcrumb (document title, section, subsection) so a piece about “cancellation fees” still says which policy and which plan it belongs to when it arrives on its own. Attach metadata: source document, section, effective date, owner, and who is allowed to see it. Let chunks overlap slightly so a sentence at a boundary appears in both neighbours.
Done well: any chunk, read alone, makes sense and says where it came from. Test it by pulling ten chunks at random and reading them cold. If you cannot tell what they are about, neither can the system.
2. Pick an embedding model and keep it
An embedding model turns a chunk (or a question) into a list of numbers representing its meaning, so similar things land near each other. OpenAI, Google, Anthropic’s partners, and several open-weight models all offer one. Differences exist but are smaller than the difference a good chunking pass makes.
The rule that matters: choose one and record which one, because every chunk must be embedded with the same model as the questions. Switch models and you must re-embed everything. Store the embeddings in pgvector (an extension to the Postgres database most business apps already use) unless you have a reason to run a separate vector store like Pinecone. Fewer systems, fewer things to break.
3. Retrieve by meaning and by keyword, then re-rank
Meaning search finds “dripping faucet” for “leaking tap.” It does badly on exact things: part numbers, invoice IDs, proper names, defined legal terms. Keyword search does those well and misses synonyms. Run both (this is called hybrid search), merge the results, and pass the top candidates through a re-ranker, a second small model that scores each chunk against the question more carefully than the first pass could.
Then filter before anything reaches the model: by permission (below), by date if the question implies “current,” by product line or client if the metadata allows it. Retrieve more than you need (say twenty), re-rank, and hand the model the best five to eight. Too few and the answer is missing context. Too many and the model gets distracted by near-misses, and you pay for every token (roughly three quarters of a word) you send.
Done well: for a set of real questions with known right sources, the right chunk is in the top five almost every time. That number, called recall, is the single most useful measure of a RAG system, and you should know yours.
4. Require a citation for every claim
The model is told to answer only from the supplied chunks and to mark each statement with which chunk it came from. The application turns those marks into links the reader can click to see the source passage. This does two things. It lets a user check any answer in seconds. And it lets you spot the failure mode where the model answers from its general knowledge instead of your material, because those sentences have no citation.
Reject uncited answers in customer-facing systems. An answer with no source is a guess, and a guess that reaches a customer is a liability. Internal tools can be softer, but should still show the sources.
5. Build the “not found” path on purpose
If retrieval returns nothing relevant, or only weak matches, the model must say so rather than improvise. This does not happen by default. Instruct the model explicitly, and check the retrieval scores in code: if the best match is below a threshold, do not even ask the model, return “I could not find that” and route the question somewhere useful (AI Chatbot Escalation to Human: When and How the Handoff Works covers the handoff).
Log every “not found.” That log is the most valuable output of the whole system, because it is a list of what your documents do not cover, written by your own users. Review it weekly and add the missing material (Knowledge Base for an AI Chatbot: What to Write and How).
6. Keep it fresh, and know how fresh it is
Documents change. The pipeline needs a way to notice: a nightly sweep, a webhook from the document system, or a “re-index” button someone presses. When a document changes, its old chunks are removed and new ones written. The old chunks must go; a system that only adds accumulates contradictions.
Every chunk carries a date. Every answer can show “based on material updated on X.” When two chunks disagree, the newer one wins by rule, and the older one should probably be retired from the source. Ask any vendor how long it takes for a corrected document to change the answers. “Minutes” or “overnight” are fine. “We can do that for you” is not.
7. Enforce permissions before retrieval, not after
Every chunk carries who may see it: public, all staff, a department, a specific client account. When a question arrives, the system knows who is asking and filters the search to what they may see, before the model sees any results. Filtering afterwards is not enough; the model may have already paraphrased the forbidden chunk into its answer.
This is the most commonly skipped part and the most expensive to skip. One store with everything in it, one chatbot in front, and an internal pricing memo ends up in a customer’s answer. Build the permission model on day one. It is a column and a filter, not a project, if it is done at the start.
8. Measure it with a golden set before and after every change
Write down fifty to two hundred real questions, the right answer to each, and the source chunk that contains it. That is your golden set. Run it whenever anything changes: the chunking, the embedding model, the prompt, the re-ranker, the underlying model. Record recall (was the right chunk retrieved) and answer quality (was the answer correct, cited, and did it admit “not found” when it should).
Without this, every change is a guess and every regression is discovered by a user. With it, a developer can try a new chunking strategy on Tuesday and know by lunch whether it helped. The full method, including using a model to grade answers, is in AI Evaluation and Evals: Testing an AI Feature Before Launch.
The cheap version most businesses should start with
Everything above can be built simply. In the systems we build, the first version is usually: documents in a folder or a wiki, structure-aware chunking, one embedding model, pgvector in the existing Postgres, hybrid search with a re-ranker, a prompt that requires citations and permits “not found,” a permission column, a nightly re-index, and a golden set of fifty questions. No separate vector database, no orchestration platform, no agent. That version answers most business questions well and every part of it can be inspected.
Add complexity only when the golden set tells you where it is needed. Bigger models, fancier retrieval, and multi-step reasoning all have their place, and the measurement tells you when you have reached it.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: a property management company with about sixty staff managing several thousand residential units across a region. Leasing agents, maintenance coordinators, and accounting all field the same questions: what does this lease say about pets, what is the procedure for a security deposit dispute in this county, which vendor is approved for HVAC in this building.
What was wrong: an earlier vendor connected a chatbot to the shared drive. It answered well in the demo. In use it mixed up the pet policy across two lease templates, quoted a deposit rule from a county the company no longer operated in, and once surfaced an internal memo about a rent increase strategy to a leasing agent who pasted it to a tenant. The company turned it off.
What gets built:
- Lease templates and procedures cut by clause, with property, county, template version, and effective date on every chunk. Retired templates are excluded rather than deleted, so historical questions still work when someone asks for them explicitly.
- Hybrid search with a re-ranker, filtered by the property and county the asker is working on, so “pet policy” retrieves the right template.
- Permissions on every chunk: leasing, maintenance, accounting, management. Internal memos are visible only to management. The filter runs before results reach the model.
- Citations on every statement, linking to the clause. A “not found” response when the best match is weak, which opens a ticket to the operations lead.
- A golden set of eighty real questions collected from staff, run before launch and after every change.
- A nightly re-index from the document system, and a “last updated” date shown on every answer.
What changes: recall on the golden set goes from about half (the old system, measured after the fact) to nearly all, and the company knows the number. Leasing agents stop asking managers about pets. The “not found” log exposes six procedures that were never written down, which get written. Nobody sees a memo they should not.
What it costs to run
Model calls for RAG are larger than plain chat because the retrieved chunks travel with every question. A business handling a few hundred questions a day, using a mid-tier model, should expect something between tens and a couple of hundred dollars a month; check the current pricing page for the model you pick. The re-ranker is cheap, usually a fraction of the answer cost. Embedding a few thousand documents costs a few dollars once, then pennies per changed document.
pgvector adds nothing to the database bill. A hosted vector store, if you choose one, has free and low tiers at this scale. The application itself runs on a small server, roughly $10 to $40 a month. The ongoing human cost is the person who reads the “not found” log and keeps the documents current, a few hours a month, and it is the cost that determines whether the system is still good in a year.
The mistakes we see most
- Fixed-size chunks. Cutting every 500 words and wondering why the answer to a two-part rule is only half right.
- Meaning search alone. Part numbers, invoice IDs, and names never match, so the system looks clever and fails on the exact questions that matter.
- No threshold, no “not found.” The model always answers, so the weakest retrieval produces the most confident nonsense.
- Permissions as an afterthought. One big store, one chatbot, one leaked memo.
- Nothing measured. No golden set, so nobody knows whether last week’s change helped or hurt.
- Old chunks never removed. The system accumulates every version of every policy and picks one at random.
When to bring in help
If your questions come from a single well-organised knowledge base and there are no permission distinctions, an off-the-shelf “ask your documents” feature in a help desk or knowledge tool may be enough. Test it with your own hardest questions and with questions it should refuse, and look at whether it cites sources.
You need a developer when the material lives in several systems, when who-may-see-what matters, when you want citations and thresholds you control, when the answers need to feed your own software (How to Add AI to Existing Software Without Breaking It), or when you want a golden set that tells you the truth. A capable developer builds the cheap version in a few weeks; most of the effort goes into chunking and measurement, not the model.
Levelbrook builds this for businesses: structure-aware chunking, hybrid retrieval, citations, permissions before the model, and a golden set you keep. Fixed price from a written scope, everything runs in accounts you own, and the form below is how a conversation starts.