Where RAG sits in a SaaS you already run
Six parts, and only one of them is the model:
- Connectors and ingestion. Pull documents from wherever they live in your product and outside it, normalise them, split them into passages.
- An index that keeps up. The reindex interval matters more than the corpus size. The system measured below indexes roughly 2,500 documents and reindexes them daily, because the data changes daily.
- A retrieval service that filters by tenant and permission first, then ranks. This is the part that decides the design, and the next section is about it.
- One interface to the model, so the provider can change without touching the product: streaming, fallback, token budgets.
- An answer surface in your product, with the sources shown, so a user can check what the answer rests on.
- An evaluation set and monitoring, so a quality drop is a failed check rather than a support ticket.
For the index, if you already run PostgreSQL, keeping vectors in it with pgvector puts them next to the data and the tenant model you already enforce. A dedicated vector store — Pinecone, Weaviate, Qdrant — earns its place when the corpus or the query load outgrows the database you have, not before.
Tenant isolation is the requirement that decides the design
In a multi-tenant SaaS, the one failure you cannot have is an answer built from another customer’s documents. The prompt is not a security boundary: telling a model to ignore passages it has been given is not the same as not giving them. So the filter goes into the retrieval query itself, by tenant and by permission, before ranking — the same way tenancy is enforced in the schema rather than in middleware, which is decision one of building a multi-tenant SaaS.
It is also why retrieval rather than fine-tuning: anything trained into a model is available to every tenant who reaches it. The longer version of that argument.
The latency budget, measured
| Stage | Time, average |
|---|---|
| Retrieval | about 0.3 s |
| Generation, to the last token | about 1.5 s |
| Everything else in the pipeline | about 0.1 s |
| Total, average | about 1.9 s |
| Total, p95 | about 4.2 s |
Measured to the last token, not the first. That is the harder number to publish and the one a user actually waits for. Streaming the answer lets the first words appear well before it; it does not change when the answer is complete.
The cost per answer, measured
About $0.011 per answer: roughly $0.002 of retrieval and $0.009 of generation, across about 40,000 queries a month — which is about $440 a month, as arithmetic. The split is the transferable part. Retrieval infrastructure is the cheap fifth; what retrieval really costs arrives in the generation line, as the extra input tokens every retrieved passage adds to every prompt. Context length is the cost lever, not the vector database. To see what that lever is worth at your own prices and volume, the AI token cost calculator prices the retrieved context as its own line.
Rolling it out without risking the product
- Behind a flag, to a few tenants first, so real usage reaches the evaluation set before every customer does.
- The golden dataset before the code: real questions with known correct answers, from a random sample of real documents rather than the tidy ones.
- The reindex schedule as a product decision: how stale may an answer be? Daily is right for a corpus that changes daily, and wrong for one that changes by the minute.
What it costs to build
An LLM feature added to a product that already exists — document Q&A is the classic case — is our $15,000–$30,000, 2–4 week band, at a fixed price agreed after a discovery call. The variable is not the model; it is how much of your existing codebase has to be touched to reach the data. How we build it is on the AI integration page; if you would rather build it inside your own team, our AI engineers join your board from $4,000 a month.