Three vendors have now told you their product “uses RAG.” One said it like it settled the matter. You nodded. You did not ask what it meant, because the demo was moving and the sales engineer clearly expected you to know.
Here is the honest situation. RAG is a real technique, it is the reason an AI system can answer questions about your business instead of about the internet in general, and it is also the most over-claimed phrase in AI sales. Some products that say “RAG” have built something careful. Some have bolted a search box to a chatbot. From the outside the demos look identical.
By the end of this article you will know what RAG is, in plain terms, using a library instead of a diagram. You will know what problem it solves, the problems it does not solve, and the three questions that separate a vendor who has done the work from one who has not.
What this actually is
Start with the thing RAG is fixing. A large language model (the system behind Claude, GPT-class models, and Gemini; the plain explanation is in What Is an LLM? A Plain Explanation for Business Owners) has read an enormous amount of public text. It has not read your price list, your policies, your contracts, or the email your operations manager sent last Tuesday. Ask it about your business and it has two options: say it does not know, or produce a fluent, plausible answer that sounds like your business. It very often does the second. That is a hallucination, a confident answer with nothing underneath it.
RAG stands for retrieval-augmented generation. Strip the jargon and it is this: before the model answers, the system goes and finds the relevant pages from your own material, puts them in front of the model, and says “answer using this.” Retrieval is the finding. Augmented means the model’s answer is topped up with what was found. Generation is the model writing the reply.
The library analogy makes it concrete. Picture a very well-read reference librarian who has never set foot in your company. If you ask her about your return policy from memory, she will guess, politely and wrongly. Now picture the same librarian, but before she answers, an assistant runs into the stacks, pulls the three pages of your policy binder that mention returns, and puts them on her desk. She reads them and answers from the pages. That is RAG. The librarian is the model. The assistant is retrieval. The binder is your knowledge base. The quality of the answer depends far more on the assistant and the binder than on the librarian.
That last sentence is the whole article. Vendors sell the librarian. What you are buying is the assistant and the binder.
How RAG works, one step at a time
1. Your material gets cut into pieces
The system cannot hand the model your entire document store every time; there is a limit to how much text a model can read in one go (its “context window,” roughly a few hundred pages for current models, and more text costs more money and attention). So each document is cut into pieces, usually a few paragraphs each. These are called chunks.
How the cutting is done matters more than anyone expects. Cut a pricing table in half and neither half makes sense. Cut a policy so the exception lands in a different chunk from the rule, and the model sees the rule without the exception. A good implementation cuts along the document’s own structure and attaches to each chunk where it came from: which document, which section, what date.
2. Each piece gets a position on a map of meaning
For the assistant to find the right pages fast, every chunk is turned into a list of numbers that represents what it is about. That list is called an embedding. The useful mental picture is a map: chunks about refunds cluster in one region, chunks about delivery times in another, and two chunks that say the same thing in different words land close together. Your question gets placed on the same map, and the system pulls the chunks nearest to it.
This is why RAG search finds “dripping faucet” when you ask about a “leaking tap.” It is matching meaning, not spelling. The embeddings live in a database that can find nearest neighbours quickly: a dedicated product like Pinecone or, for most businesses, the pgvector extension inside the Postgres database the application already uses.
3. The question goes in, the best few chunks come out
When a user asks something, the system embeds the question, finds the closest chunks (often five to twenty), and usually runs a keyword search alongside, because meaning search alone misses exact part numbers and names. The results are ranked and the top few selected. This is where most bad RAG systems fail: they retrieve the wrong chunks, or too many, or chunks the user was never allowed to see.
4. The model answers from the chunks, and says where it looked
The selected chunks are placed in front of the model with an instruction along the lines of: “Answer the question using only the material below. If the material does not contain the answer, say so. Cite which document each fact came from.” The model writes the reply. A good system shows the citations to the user, so a wrong answer can be traced to its source in seconds.
That is all RAG is. Cut, map, find, answer. Everything a vendor adds on top is refinement of those four steps, and the refinements are where the quality lives (RAG Implementation Best Practices That Actually Hold Up covers them one by one).
What RAG fixes and what it does not
5. It fixes “the model does not know my business”
This is the main event. With RAG, the model can answer from your documents, tickets, catalogue, and policies without anyone retraining it. Update the binder and the answers update. That is a large advantage over the alternative (fine-tuning, which means teaching the model new habits by training it on examples: expensive, slow to change, and poor at teaching it facts anyway).
6. It reduces invented answers, but only if the retrieval is good
RAG makes hallucination less likely because the model has something real to work from. It does not make it impossible. If the assistant brings back the wrong pages, the librarian answers confidently from the wrong pages. If the binder is out of date, the answer is out of date. If nothing relevant is found and the system is not told to admit that, the model fills the gap with something plausible. RAG moves the problem from “the model is guessing” to “is the retrieval right,” which is a better problem because you can measure it. It is still a problem.
7. It does not fix a bad knowledge base
If your policies contradict each other, RAG will faithfully retrieve both versions and the model will pick one. If the answer to a common question lives only in the head of your longest-serving employee, RAG cannot find it. If your documents are scanned PDFs of handwritten forms, there is nothing for the map to place. The quality ceiling of any RAG system is the quality of the material behind it, which is why Knowledge Base for an AI Chatbot: What to Write and How argues the knowledge base is the real product.
8. It does not understand permissions unless someone builds them
The assistant will retrieve any page it can reach. If your HR policies and your customer FAQ are in the same store, a customer asking a clever question can get an HR answer. A properly built system attaches permissions to every chunk and filters before the model ever sees the results. Ask about this. It is the most common thing left out.
The three questions to ask a vendor who says “we use RAG”
9. “Show me the citation on a wrong answer”
Ask them to make the system produce an answer you know is wrong, or find one in their demo, and then show you where the answer came from. A real implementation shows the exact chunk and document. A weak one shows you a link to the whole PDF, or nothing. If they cannot trace an answer to its source, you cannot either, and you will find that out from a customer.
10. “What happens when the answer is not in my documents?”
Type a question the material cannot answer and watch. The right behaviour is a plain “I could not find that in the available material” and, in a customer-facing system, a route to a person. The wrong behaviour is a fluent paragraph. Every RAG system will produce the wrong behaviour unless someone deliberately built the right one.
11. “How does it know who is allowed to see what, and how fresh is it?”
Two questions, one breath, because the answers reveal the same thing: whether they built a system or a demo. On permissions, you want to hear that each piece of content carries who may see it, and that the filter happens before results reach the model. On freshness, you want to hear how a changed document gets re-cut and re-mapped, and how long that takes. “We re-index nightly” is a fine answer. “It just works” is not.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: a commercial insurance brokerage with twenty-two staff. The producers sell, the account managers service, and everyone spends a large part of the day answering the same questions: what does this carrier’s policy form say about water damage, what is our procedure for a mid-term cancellation, which endorsements does this client have. The answers live in a few hundred carrier PDFs, an internal procedures wiki that is mostly current, and email.
What was wrong: new account managers took months to become useful because the knowledge was scattered. Experienced staff were interrupted constantly. A vendor demoed a chatbot that “used RAG,” answered the demo questions beautifully, and, when one manager typed a real question, invented a coverage exclusion that did not exist in that carrier’s form.
What gets built instead:
- A cleanup pass on the procedures wiki, done by the staff, so there is one current version of each procedure with a date on it. Nothing technical; this is the binder.
- Carrier forms cut by section, with the carrier name, form number, and effective date attached to every chunk, stored in pgvector inside the brokerage’s existing database.
- Retrieval that combines meaning search with keyword search, so form numbers and defined terms match exactly, and that filters by which carrier the client is actually with.
- A model instruction that requires a citation for every statement and a plain “not found” when the material does not answer, with a one-click “ask a senior” button that opens a ticket with the question and what was retrieved.
- A weekly report of the questions that got “not found,” which becomes the list of things to add to the wiki.
What changes: new staff get correct, cited answers to procedure questions on day one. Senior staff get interrupted less and, when they are, the question arrives with the relevant form section already attached. The “not found” report turns the knowledge base into something that improves every week instead of rotting.
What it costs to run
The model calls are billed by usage in tokens (roughly three quarters of a word). RAG requests are larger than simple ones because the retrieved chunks go in with the question, so expect each answer to cost more than a plain chat reply, and expect a business with a few hundred questions a day to spend somewhere between tens and a couple of hundred dollars a month on a mid-tier model. Check the current pricing page; the numbers move.
Embedding your documents is a one-time cost, typically a few dollars for a few thousand pages, and then pennies as documents change. Storage is cheap: pgvector runs inside Postgres you already pay for. A small server for the application itself runs roughly $10 to $40 a month.
The real cost is the binder. Someone has to own the documents, keep them current, and read the “not found” report. That is hours a month, every month. A RAG system without that person decays quietly.
The mistakes we see most
- Buying the librarian. Comparing vendors on which model they use, when the difference in answer quality comes almost entirely from retrieval and the knowledge base.
- Skipping the cleanup. Pointing RAG at a shared drive with four versions of every policy and expecting it to pick the right one.
- No “not found” path. The system always answers, so nobody ever learns what it could not find.
- One big store with no permissions. Internal documents leaking into customer-facing answers because retrieval had no idea who was asking.
- Never measuring retrieval. Nobody writes down fifty real questions with known right answers and checks how often the right chunk is retrieved. So nobody knows the system is at sixty percent until a customer tells them (AI Evaluation and Evals: Testing an AI Feature Before Launch).
When to bring in help
You can try RAG yourself with off-the-shelf tools: several help-desk and knowledge products have a “connect your documents and ask questions” feature, and it is worth an afternoon to see how your material behaves. If the answers are good on your real documents, not the vendor’s, you may not need anything custom.
You need a developer when the material is spread across systems, when permissions matter, when you want citations and a “not found” path you control, or when the answers need to flow into your own software. A competent build for a business this size is measured in weeks, and most of those weeks go on retrieval quality and the knowledge base, not the model.
Levelbrook builds this for businesses: retrieval you can inspect, citations on every answer, permissions before the model, and a report of what could not be found. Fixed price from a written scope, everything runs in accounts you own, and the form below is how a conversation starts.