Why a demo is not an evaluation
Most RAG demos are built on the documents that make it look good: clean PDFs, one language, the questions the builder already knows the answers to. Production sends everything else. On the logistics platform the documents came from 14 disconnected sources — invoices, customs paperwork, shipment documents, email attachments, internal exports — in five operating languages, against a knowledge base of about 2,500 documents and 15,000 pages that was reindexed every day.
An evaluation answers two questions a demo cannot. How often is the output correct on the documents this business actually receives? And is it still correct after the prompt, the model, the chunking or the retrieval settings change? Without the first, nobody can say whether the system is good enough to replace manual work. Without the second, every improvement is a guess that might have broken something else.
Build the golden dataset from your real documents
A golden dataset is a set of inputs with the correct output already agreed by people. On the logistics build it was about 2,000 documents, built across the languages and document types the system had to handle, and it existed from week one — before the extraction flows were shipped, so every version of them could be measured against the same thing.
The labels were made by our engineers together with the client’s analysts, and that pairing is the point. Engineers know what the pipeline sees in a document; analysts know what “correct” means for the business. Where two careful people disagree about a label, the disagreement is itself a finding: a definition of correct that people read differently will be read differently by a model too.
Three rules hold for any golden set, ours or not:
- Size it by variety, not by volume. Every source, language and document type the system will meet needs to be in it. A thousand clean invoices from one supplier measure one supplier.
- Sample, do not select. Pick documents at random from the real flow, including the scans, the partial ones and the odd formats. The tidy examples are the ones a demo already handles.
- Freeze it, then add to it deliberately. A golden set that changes between two runs cannot tell you whether the system or the test moved. New cases go in as a versioned addition, never as a silent edit.
RAG evaluation metrics: retrieval and generation separately
A wrong answer from a RAG system has two possible causes: retrieval did not find the right material, or the model did something wrong with material it had. One overall score hides which one it was, so the standard practice is to measure the layers apart.
| Layer | Question it answers | Typical metric | Needs a human label? |
|---|---|---|---|
| Retrieval | Was the right source in the context the model saw? | Recall or hit rate at k; rank of the first correct source | Yes: which source holds the answer |
| Grounding | Is every claim in the output supported by that context? | Faithfulness, groundedness | Can be checked against the context itself |
| Answer | Does the final answer match what a person agreed is correct? | Correctness against the label | Yes: the label |
| Extraction | Is each value written to the target system right? | Field-level accuracy, per field type | Yes: the correct value per field |
| Confidence | Does low confidence really mean a likely error? | Accuracy above and below the review threshold | Yes, to calibrate the threshold |
| Operations | Is it fast and cheap enough at real volume? | Latency (average and p95), cost per answer | No |
On the logistics platform the headline metric was field-level accuracy, because the output was not prose but values written into the client’s ERP: a wrong date or amount is the error the business pays for. The operating numbers were measured the same way: 1.9 seconds on average to the last token (4.2 s at p95), and about $0.011 per answer across roughly 40,000 queries a month. Running that cost for your own volume is what the LLM cost calculator is for.
Field-level vs document-level accuracy
The 99.2% on our case study is field-level: each extracted value is counted on its own, and 99.2% of them were correct. That is the honest unit for this system, but it is not the same number as “documents processed without an error”, and an evaluation report has to say which one it gives.
Arithmetic, not a measurement from the project. If a document carried 20 fields and errors were independent, 99.2% per field would leave about 85% of documents with every field right (0.99220 ≈ 0.85). Real errors cluster — one bad scan breaks several fields at once — so the true document-level figure is usually higher than that, but the gap is why a vendor quoting “99% accuracy” without a unit has told you very little.
Report both, and per field type where it matters. An average across field types can hide a weak one: easy fields such as dates pull the total up while a hard one, such as a free-text description, sits well below it.
Route uncertain outputs to people
No extraction system is right every time, so the useful question is which outputs a person should see. On the logistics platform every value carried a confidence score, and values below the threshold went to an analyst review workspace with the source highlighted and every decision kept in an audit trail. After all the improvements, about 4% of fields were routed there.
That is how 87% less manual processing and human control live in the same system: the analysts’ work moves from keying values to checking the small share the model is unsure about, with the source highlighted beside it.
The threshold is a business decision as much as a technical one. Set it lower and more values reach people, so review costs more. Set it higher and more errors go through unseen. The golden set is what makes the choice measurable: it shows the accuracy of the values above the threshold and of those below it, so the trade can be priced instead of guessed.
Make evaluation a release gate
On the logistics build the golden set was re-run on every change to prompts, models or the pipeline, and a drop below the accuracy threshold blocked the release. That turns evaluation from a report somebody reads into a test the deployment has to pass, the same way unit tests stop a build. Four things make such a gate trustworthy:
- Pin what is being compared. The same frozen golden set, the same scoring code, results stored per run, so two runs can be compared field by field.
- Gate on the number the business relies on. For an extraction system that is field-level accuracy; for an answer system, correctness and grounding together.
- Look at slices, not only the total. A change that improves one language and breaks another can leave the average unchanged.
- Treat a model upgrade as a change. A newer model is not automatically better on your documents, and the gate is how you find out before your users do.
The gate covers changes to the system. Changes to the data — a new supplier, a new document format — arrive without a release, which is why the platform also went live with monitoring and accuracy dashboards in production.
What the score does not tell you
- It is not a comparison with people. 99.2% is what this system scores against the labels. Nobody measured the manual process before it was replaced, so there is no comparable “before” figure, and the case study does not claim the system is more accurate than the people it took work from.
- It ages. A golden set describes the documents of the month it was built. New suppliers and formats need new labelled cases, or the score stays high while production drifts.
- An LLM judge is not ground truth. A model scoring another model is a useful second check on open-ended answers. For values that end up in a system of record, the labels have to come from people, and the judge has to be checked against them.
What evaluation costs, and when to start
Evaluation is part of the build, not an add-on after it. On our published bands, a full AI product — retrieval pipeline, frontend, backend, integrations and the evaluation work a demo does not need — runs $40,000–$120,000 over 8–12 weeks. The logistics platform went live in 10. An LLM feature added to a product you already run is $15,000–$30,000 over 2–4 weeks, and it needs a smaller golden set, not none.
The cheapest moment to start is before anything is built. A data audit on a genuine random sample of your documents tells you whether a golden set can be agreed at all, and it is the first step of AI consulting by the hour. Where it leads next: custom AI document processing when the output is values, AI integration when it is answers inside your product, and the trade-off in RAG or fine-tuning if retrieval is not the right tool at all.