You saw the demo. The vendor typed three questions, the AI answered all three beautifully, and everyone nodded. Then it went live, and within a week your office manager forwarded a screenshot of it telling a customer you offer a service you have never offered.
Nobody lied to you. The demo was real. The problem is that three good answers tell you nothing about the next three thousand, and a feature that is confidently wrong some of the time reaches customers with your name on it.
There is a plain discipline for this. In software it goes by the name AI evaluation, or evals for short. This article explains what an eval is, how to build one from cases you already have, how to grade results without reading every answer yourself, and the go-live gate that decides whether a feature ships.
What an eval actually is
An eval is a test for an AI feature: a set of inputs where you already know what a good answer looks like, run through the AI, with a count of how many it got right.
The analogy is a new estimator on probation. You do not judge them on the first quote they write. You give them twenty jobs you have already priced, compare their numbers to yours, and look at where they drifted. That is an eval: a fixed set of known cases, a rule for what counts as right, and a score.
Ordinary software does the same thing every time, so its tests are simple. An AI feature built on a large language model (an LLM, the kind of system behind Claude, GPT-class models and Gemini) does not. Ask it the same question twice and you may get two differently worded answers. Change one line of the instructions and unrelated answers can shift. Wait three months and the provider will have updated the model underneath you. So the test has to run repeatedly, against enough cases that a few odd results cannot hide a real problem.
The three parts of any AI evaluation are the golden set (the cases and their known-good answers), the grader (the rule that decides pass or fail), and the gate (the score below which you do not ship).
AI evaluation practices that separate a tested feature from a demo
1. Build the golden set from real cases, not invented ones
The golden set is the collection of inputs and expected answers that every test run uses. The biggest mistake is writing it from imagination. Invented “typical customer questions” are polite, well spelled and one-topic. Real customers write in fragments, ask two things at once, and call your product by a nickname only your staff know.
Pull the cases from what you already have: the support inbox, call transcripts, quote requests, website forms. Strip anything personal, and keep the awkward ones. The customer who asked “do u do the thing with the pipes” is exactly who the feature meets on day one.
Aim for fifty to three hundred cases to start; fewer than about thirty and a lucky run will fool you. Group them by type (pricing, hours, complaints, out-of-scope) so the score can be read per group.
2. Write the expected answer as a checklist, not a script
For each case you need a description of what a good answer must contain and what it must not. Do not write the exact answer you want; the model phrases things differently every time.
Write a checklist instead. “Must state we are closed Sundays. Must not invent a Sunday phone line. Must offer the booking link.” For a drafting feature: “Mentions the delayed part, apologises once, gives the new date, does not promise a refund.”
The must-not list matters more than the must list. In the systems we build, the failures that hurt are almost never a missing detail. They are an added one: a price the model was never given, a policy it made up, a promise nobody authorised. A must-not violation is an automatic fail regardless of how good the rest of the answer is.
3. Grade with hard rules first, and a judge model second
Some grading is done by simple checks that never get tired. Did the answer include the booking link. Is the extracted total within one cent of the known total. Does it contain a forbidden phrase. These rule checks are cheap and reliable and should carry as much of the grading as they can.
Everything needing judgment (was the tone right, did it actually address the question) is usually graded by a second model. This is LLM-as-judge: you give a capable model the question, the answer and your checklist, and ask it to score and explain. It is far cheaper than a person reading every result and, with a concrete checklist, consistent enough to be useful.
It has real limits. A judge can be fooled by fluent wrong answers, tends to prefer longer ones, and can be lenient on its own provider’s output. Keep its checklist concrete (“does the answer state the Sunday closure: yes or no”) rather than vague, and spot check it: once a month, pull twenty verdicts at random and have a person confirm them. Disagreement above one in ten means the checklist needs tightening.
4. Set the pass and fail rules before you see the results
Decide the gate in advance. Set it after the first run and you will set it wherever the feature happened to land. Write down something like: “Overall pass rate at least 95 percent. Zero must-not violations in the pricing and legal groups. No group below 85 percent.”
Those are examples, not a standard. The right bar depends on what a wrong answer costs. Anything touching money, health or a legal obligation should have a must-not violation rate of zero, and the fallback for anything uncertain should be a handoff to a person (AI Hallucination Guardrails for Business Applications covers how that is built). A fail is not a crisis, it is information: fix the instructions or the material and run again. What must never happen is quietly lowering the bar to make the number green.
5. Run the whole set on every change, including changes you did not make
An AI feature has three moving parts: the prompt (the written instructions the model works from, see Prompt Engineering for Business Applications That Work), the material it retrieves (your knowledge base, your product data), and the model itself. Change any one and every answer can shift.
So the eval runs on every change. Edited one sentence of the prompt to fix complaint handling. Run the full set, because that sentence may have made pricing answers worse. This is regression testing, trivial by machine and impossible by hand, which is why the eval must be a script anyone can run with one command.
Run the eval through the whole pipeline, too: if the feature looks up your knowledge base before answering (the pattern called RAG, explained in RAG Implementation Best Practices That Actually Hold Up), the eval starts from the raw question and goes through the lookup, not from a hand-picked excerpt.
The change you did not make is the model update. Providers retire versions on their own schedule, and the new one may be worse on your cases in some specific, surprising way. Pin the exact model version, run the eval against a candidate when one appears, and switch only when the score holds. The same eval is how you compare providers fairly (Choosing an LLM Provider for Your Business in 2026).
6. Keep a set of things that should be refused
Ten to twenty percent of the golden set should be inputs where the correct behaviour is to decline, hand off or ask for clarification. Questions outside your business. A discount the feature is not authorised to give. Attempts to make it ignore its instructions (Prompt Injection Explained: AI Security for Your Business covers that angle).
Grade these strictly. An answer that helpfully invents a discount policy is a fail even if it sounds gracious. “I can’t help with that, but I’ve passed it to the team” is a pass. This is the part vendors are least likely to show you in a demo.
7. Read the failures, and show the owner
The score is the headline; the failures are the story. After every run, someone reads every failed case and sorts it: the prompt was unclear, the material was missing or wrong, the model made something up, the grader was wrong. Each has a different fix. When a customer catches the feature saying something wrong, that exchange (cleaned of personal details) becomes a permanent case in the golden set, and it can never come back silently.
The eval should also produce a one-page report a non-technical person can read: pass rate overall and per group, must-not violations, and the ten worst cases with the actual answer beside the expected one. In our experience an owner who reads that page monthly spots problems the developer misses.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: a regional HVAC company with eleven technicians and two office staff, adding an AI assistant to its website to answer questions and book service calls. The vendor’s demo answered “how much is a tune-up” perfectly. The owner, burned once by a chatbot that quoted an old price for three months, asked to see the evals. There were none.
Before launch, the office staff pulled 140 real questions from two months of website chats and email and wrote a short checklist for each. Thirty were cases where the right move was a handoff: gas smells, warranty disputes.
What got built:
- A script that runs all 140 cases through the live feature, through the same knowledge lookup customers hit, and saves every answer.
- Rule checks for hard facts: prices matching the current list, the booking link present when a service was discussed, forbidden phrases absent.
- A judge model grading tone and completeness against each checklist, with its reasons saved.
- A gate: 95 percent overall, zero pricing or safety violations, no group below 85.
- A one-page report emailed to the owner after every run.
The first run scored 81 percent. Eleven failures were the assistant confidently quoting a maintenance plan the company had discontinued; the knowledge base still had the old page. Six were gas-smell questions answered with troubleshooting steps instead of “leave the house and call this number.” Both were fixed in a day and the third run passed. A month later the provider released a new model version, one group dropped four points on it, and the switch waited until the prompt was adjusted. None of this reached a customer.
What it costs to run
AI evaluation itself is cheap. Running 200 cases through a current model costs a few cents to a few dollars per run, depending on answer length; check your provider’s pricing page for per-token rates (a token is roughly three quarters of a word, billed per million). The judge roughly doubles that. Running the eval on every change and weekly on a schedule should cost tens of dollars a month, not hundreds.
The real cost is time. Building the first golden set takes someone who knows the business between a day and a week. Reading the failures after each run takes an hour or two. The script and report are a developer job of a few days at the start. Hosted eval products exist and are worth a look once the loop is running; a script, a folder of cases and a spreadsheet is a complete eval to start.
The mistakes we see most
Testing on the demo questions. The vendor’s five questions become the test set. The feature passes them forever and fails on everything else.
No must-not rules. Every expectation says what the answer should contain and nothing says what it must never contain. The feature is graded a success while inventing policies.
Grading only by the judge. A judge with a vague checklist passes almost anything fluent.
Running it once. The eval passes before launch and never runs again. Three months later the model has been updated twice and nobody knows what the feature says.
Moving the bar. The score comes in at 88, the gate was 95, and the gate becomes 85. The number is green and the feature is exactly as bad as it was.
When to bring in help
If you use an off-the-shelf AI product, you can run a meaningful eval yourself with no developer. Pull fifty real cases, write the checklists, paste each into the tool, and grade by hand in a spreadsheet. It takes an afternoon and it is more than most buyers ever do. Ask the vendor for their eval results too; one who cannot answer has not tested the product against anything like your business.
Once the feature is custom, or touches your own data, AI evaluation needs a script that runs the full set through the real pipeline, grades it and reports needs a developer. So does pinning model versions and wiring the eval into deployment so an untested change cannot ship. That is a few days of work, and it is what turns a demo into something you can stand behind.
Levelbrook builds AI features for businesses with the eval as part of the delivery: the golden set from your cases, the grader, the gate and the report, all running in accounts you own. Fixed price from a written scope. If you would like to talk through a feature you are considering, the form below is how the conversation starts.