You have seen the AI write a paragraph. What you want is for it to fill in the form. Pull the invoice number, the amount, and the due date out of that email and put them in the right fields. Decide which of your six queues a ticket goes in. Look up an order in your system and tell the customer where it is. The paragraph is a party trick; the form is the business.
The gap between the two is what this article is about. A model produces text. Your software needs data it can trust: fields, types, allowed values. Bridging that gap is called structured output, and letting the model reach into your systems to look things up or do things is called tool calling. Both are standard features of every major provider, and both come with one rule an owner must understand: the model proposes, your software disposes.
By the end you will know what structured output from an LLM is, what a schema is and why it matters, how tool calling works, what validation means and why it is not optional, and exactly where a human or a hard-coded check has to sit before anything irreversible happens.
What this actually is
A large language model (an LLM, the system behind Claude, GPT-class models, and Gemini; explained in What Is an LLM? A Plain Explanation for Business Owners) produces text, one piece at a time, that looks like the text it has seen. Left to itself it will write prose. Structured output is asking it to write, instead, a precise machine-readable format, usually JSON (a plain-text way of writing labeled fields and values that every programming language reads natively), and, with current providers, forcing it to match a template you supply.
That template is a schema: the list of fields, what type each one is (a number, a date, one of four allowed words), and which are required. The analogy is a paper form with boxes. Prose is a letter; a schema is the form. You would not let a new hire “just describe” an order in a letter and then have someone else transcribe it into the system. You would hand them the form. The schema is the form, and modern providers will refuse to let the model color outside its boxes.
Tool calling is the second idea. Instead of only answering, the model can say “I need to look up order 4471” or “create a ticket with these fields,” in a structured format your software recognises, and your software then does the looking up or the creating and sends the result back. The model never touches your database. It fills in a request form; your code decides whether to act on it. That decision point is where all the safety lives.
The eight rules for doing this well
1. Define the schema before you write the prompt
Start with the fields your software actually needs, their types, and their allowed values. Six fields with strict types beats twenty with “string.” A category field is a fixed list, never free text. A date is a date, in one format. An amount is a number with a currency alongside it. Mark what is required and what may be empty.
The schema is the contract between the model and the rest of your system, and writing it first exposes fuzzy thinking. If nobody can say what “priority” means in three fixed values, the model certainly cannot. Give every field a one-line description inside the schema; the model reads those and they do more work than the prompt.
2. Use the provider’s structured-output feature, not “please reply in JSON”
Every major provider now offers a mode where you supply the schema and the model’s answer is constrained to match it. Use it. Asking politely in the prompt for JSON works most of the time, and “most of the time” is a broken feature at three in the morning when the model adds a friendly sentence before the braces.
With the constrained mode the shape is guaranteed. The values are not. A model can return a perfectly formed date that is wrong. Guaranteeing the shape is what lets you spend your attention on the values.
3. Validate in code, always, even when the shape is guaranteed
The moment the answer arrives, your software checks it against the schema and against reality. Is the amount positive. Is the due date after the invoice date. Does the customer number exist in your system. Is the category one of the allowed six. Does the total equal the sum of the lines.
A validation failure is not a crash; it is a routing decision. The record goes to a person with a note saying which check failed. In the systems we build, the validation layer is where most of the engineering effort goes, because it is the thing that turns a model’s guess into data the business can rely on. The model’s job is to fill the form well. Validation’s job is to never trust that it did.
4. Ask for a confidence field and a reason, and use them
Add two fields to almost every schema: how confident the model is (a fixed scale, such as high, medium, low) and a one-sentence reason or the exact source text it used. Route anything below high to review. Show the reason next to the field so the reviewer can check in one glance.
This turns “the model got it wrong” from a mystery into a queue. The low-confidence items are also your best training material: they are the awkward cases, and they tell you what the prompt or the schema is missing (Prompt Engineering for Business Applications That Work).
5. Give the model tools with narrow, boring names
Tool calling means describing to the model a set of functions it may ask for: look_up_order(order_number), create_ticket(subject, body, queue), get_stock_level(sku). Each has a name, a description, and a schema for its inputs. The model decides which to request and with what; your code runs it.
Make each tool do one small thing, name it plainly, and give it the smallest inputs that work. “look_up_order” is good. “do_customer_thing” is not. A tool that takes free-form SQL is a disaster waiting for a customer to type something clever. The model is choosing from a menu you wrote; keep the menu short and each dish harmless.
6. Split tools into read, propose, and act, and treat each differently
Read tools (look up, search, check) are safe to let the model call freely, subject to permissions. Propose tools (draft a reply, prepare an invoice, suggest a refund) create something a person will review. Act tools (send, charge, refund, delete, change price) do something irreversible or costly.
Read tools run automatically. Propose tools run automatically and land in a review queue. Act tools either require a human click, or run only inside hard-coded limits (refund up to a fixed amount to a verified customer on a verified order, once) with every call logged. This is the trust ladder, and AI Agent Permissions: Least Privilege for Business AI goes deeper on the permissions around it. The mistake is treating all three alike. Most businesses should ship read and propose tools first and add act tools one at a time, each with its own limit.
7. The model’s request is a proposal, never an instruction
When the model returns “call refund_customer with amount 240,” that is the model filling in a form. Your software then checks: is this customer real, is this order theirs, is 240 within the allowed limit, has this order already been refunded, is a person required to approve at this amount. Only after every check passes does anything happen. “The model said so” is never the last word before money moves, data is deleted, or a message leaves the building.
This is not distrust of the model. It is the same discipline you apply to a new employee with a company card: a limit, a log, and a second signature above a threshold. The checks are where the judgment lives.
8. Log the proposal, the checks, and the outcome, together
Every structured output and every tool call is logged with what the model was shown, what it returned, which validations ran and how they came out, and what the software did as a result. Keep it for months. When a customer asks why they were charged, or a supplier asks why an order was placed, the answer is one lookup. This log is also what an auditor, an insurer, or a lawyer will ask for, and it is far cheaper to have than to reconstruct.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: a building-supplies distributor with thirty-five staff and a custom ordering system. Customers email purchase orders as PDFs and in the body of emails, in every layout imaginable. Two people spend most of their day retyping them.
What was wrong: a first attempt asked a model to “extract the order details” and paste the text into a notes field. Half the time it worked; the rest of the time a person still retyped everything, and once a quantity of 10 became 100 with nobody the wiser until the truck arrived.
What gets built:
- A schema for a purchase order: customer reference, PO number, delivery date, delivery address, and a list of lines with SKU, description, quantity, unit price, and a confidence for each line, plus a field for the exact text the model took each value from.
- Extraction on a mid-tier model in the provider’s constrained mode, so the shape is always right.
- Validation in code: every SKU must exist in the catalogue, quantities must be whole numbers within that customer’s historical range, the address must match a known site for that customer, and any line below high confidence flags the whole order for review.
- A read tool, look_up_customer, so the model can resolve “Northside job” to a delivery site, and a propose tool, draft_order, that creates an order in a “pending review” state. No act tools; a person confirms every order with one click, seeing the extracted values beside the source text.
- A log of every extraction with its checks, kept for a year.
What changes: most orders arrive pre-filled and correct, and the two staff spend their day confirming rather than retyping. The 10-becomes-100 mistake cannot recur because the quantity check flags it. Later, once months of logs show near-perfect accuracy for a few large customers, the owner considers letting those customers’ orders auto-confirm under a value limit, with a nightly report. That is an act tool, added deliberately, with a limit.
What it costs to run
Structured output and tool calling are billed like any other model usage, by tokens (units of text, roughly three quarters of a word). A schema and tool descriptions add a few hundred tokens to each request. For a business processing a few thousand documents or requests a month on a mid-tier model, expect tens of dollars, with small models handling classification-type schemas for less. Check the current pricing page; the numbers move.
The validation and logging layers run in your own software on the server you already have. The real cost is the reviewer’s time for the flagged items, which starts high and falls as the prompt and schema improve. The invisible cost, and the one to budget for, is the developer time to write good validation rules, because that is where the safety is.
The mistakes we see most
- Prose into a notes field. Asking for “the details” and getting a paragraph that a human then retypes anyway.
- JSON by request instead of by constraint. Works in the demo, breaks the night the model adds a preamble.
- Shape checked, values not. A well-formed answer with a quantity of 100 that nobody sanity-checked against the customer’s history.
- One giant tool. A single “run_query” function that lets the model do anything, including things nobody imagined.
- Act tools on day one. Refunds, sends, and deletes wired straight to the model’s say-so, without a limit or a click.
- No log. The first dispute arrives and the answer to “why did this happen” is a shrug.
When to bring in help
If your need is simple extraction into a spreadsheet, several off-the-shelf document tools do a credible job and are worth trying first (AI Document Processing for Business, From PDFs to Clean Data covers that space). If you are using an automation platform like n8n or Make, both can call a model with a schema and check fields afterwards, and a careful owner can build a read-and-propose flow without a developer.
The moment a tool can act, or the data flows into your own system of record, you need a developer to write the schema, the validation rules, the trust ladder, and the log. This is ordinary engineering, and a good developer will insist on the validation layer before you ask. If someone proposes wiring an act tool straight to the model, or cannot show you the log, get a second opinion.
Levelbrook builds this for businesses: schemas first, constrained output, validation in code, read-propose-act tiers, and a log of every decision. Fixed price from a written scope, everything runs in accounts you own, and the form below is how a conversation starts.