RAG or fine-tuning: the test that settles it
Ask one question about the failure you are trying to fix:
Is the model getting it wrong because it lacks information, or because it is responding in the wrong way?
Lacking information is a retrieval problem: it does not know your pricing, your policies, last quarter's numbers, this customer's history. No amount of weight adjustment reliably installs facts that change weekly, and attempting it produces a model that is confidently out of date.
Responding in the wrong way is a behaviour problem: the output format drifts, the tone is off, a domain-specific classification is inconsistent no matter how the prompt is written. That is what fine-tuning is for.
Most teams asking the question have the first problem and are being sold the second solution.
One production corpus, for scale. The multilingual platform behind the 87% and 99.2% figures published on this site indexes roughly 2,500 documents — about 15,000 pages, or 10–15 million tokens — drawn from 14 sources in 5 languages, and it is reindexed daily. The reindex interval is the part that decides the architecture, not the size: a corpus that moves every day cannot be fine-tuned into a model at any sensible cost, because each training run is out of date the morning after it finishes. Client anonymised.
RAG vs fine-tuning, side by side
| Retrieval (RAG) | Fine-tuning | |
|---|---|---|
| Changes | What the model knows | How the model behaves |
| Data freshness | Immediate — reindex and it is current | Frozen at training time; new facts need a new run |
| Upfront cost | Lower; the work is in the data pipeline | Higher; needs a curated, labelled example set |
| Per-request cost | Higher — retrieved context consumes tokens | Lower — shorter prompts, potentially a smaller model |
| Attribution | Natural: you can cite the retrieved source | None: the answer has no traceable origin |
| Access control | Enforceable at retrieval time, per user | Not enforceable — training data is baked in for everyone |
| Debuggability | You can inspect what was retrieved | Largely opaque |
The access-control row decides more enterprise architectures than the cost rows do. If different users are allowed to see different documents, retrieval can enforce that at query time and fine-tuning fundamentally cannot — anything in the training set is available to everyone who can reach the model.
What the cost row is worth, measured. On a production retrieval system of the size described above: about $0.011 per answer — roughly $0.002 of retrieval and $0.009 of generation — across about 40,000 queries a month. Read the split carefully, because it is easy to read backwards: the retrieval infrastructure is the cheap fifth, and what retrieval costs you sits inside the generation line, as the extra input tokens every retrieved passage adds to every prompt. Context length is the cost lever; a cheaper vector database is not.
And what it costs in time. The same system answers in about 1.9 seconds on average, 4.2 at p95 — roughly 0.3s of retrieval and 1.5s of generation, the remaining tenth elsewhere in the pipeline. The condition carries the claim: that is measured to the last token, not to the first. Time-to-first-token on the same system is a much smaller number and it is the one usually quoted, which is worth knowing before you compare anybody's latency figure with anybody else's. Your numbers move with model choice, context length and volume — the shape of them does not. Client anonymised.
When fine-tuning beats retrieval
- A strict output contract. Same structure, every time, at scale, where prompt-level compliance keeps slipping.
- Domain classification. Labels specific to your business that a general model gets subtly and persistently wrong.
- Volume economics. At high request volume, a smaller fine-tuned model can beat a large general one on both latency and cost — a real and underrated reason, and one that only applies once you have the volume. The system measured above runs about 40,000 queries a month, which is a useful marker for where that threshold is not: at that rate the per-request saving is too small to pay for a training run, let alone the repeated runs a corpus that changes daily would demand. On the latency half of the same trade, that system answers in about two seconds to the last token — the question is whether that is too slow for your users, not whether a smaller model would in principle be faster.
- House voice. Where consistency of tone is the product rather than a nicety.
What these have in common: none of them is about the model knowing things.
What to try first, and in what order
- Prompt properly first. A surprising share of "we need fine-tuning" turns out to be a prompt that never got a serious second pass. It is the cheapest experiment available.
- Build retrieval and measure. Get grounded answers with citations, then score them against a real test set rather than reading a few and forming an impression.
- Look at what is still failing. If failures are factual, the fix is in retrieval — chunking, ranking, permissions, coverage. If failures are behavioural and stubborn, you now have a labelled example set, because the failures themselves are the training data.
- Fine-tune against that measured gap, and only that.
Teams that run this sequence usually stop after step two or three. That is a good outcome, not a failed experiment — it means the cheaper architecture was sufficient, which is exactly what you wanted to find out.
The step missing from that list, and from most projects: measure the thing you are replacing, before you replace it. The system quoted throughout this article scores 99.2% because a harness was built to score it. Nobody measured the manual process it took over, so there is no comparable before — and rather than put a number from a different method next to it, we publish neither. What can be stated is the workload: 87% less manual handling. Whether the machine is more accurate than the people were is a question this project can no longer answer. Baselining costs an afternoon beforehand and cannot be bought back afterwards at any price.
What actually improves answer quality
Once retrieval is in place, quality improvements come from unglamorous places rather than from a better model:
- Chunking strategy. How documents are split determines what can be found. This is the highest-leverage knob and the least discussed.
- Ranking. Getting the right five passages into context beats getting twenty adequate ones, and costs fewer tokens.
- Coverage gaps. Frequently the corpus simply does not contain the answer, and no pipeline fixes that.
- Evaluation. Automated scoring against a fixed test set, so a regression is a failed build rather than a support ticket.
All four are part of how we scope AI development, and they are the reason the cost breakdown in what an AI SaaS product costs puts data preparation as the largest single block rather than the pipeline itself.