RAG as a Service: What Are You Actually Buying?

RAG stands for retrieval-augmented generation, which is a terrible name for a simple idea: instead of asking a language model what it knows, you first find the relevant passages in your own documents, hand those to the model, and ask it to answer using only those. The model stops guessing from memory and starts answering from your material.

Bought as a service, you are paying somebody to build and run that loop over your documents. Whether that is worth anything depends almost entirely on one thing that vendors rarely lead with, and it is not the model.

Here is what matters most:

• The retrieval step is the product. If it finds the wrong passages, a better model just writes a more fluent wrong answer.
• Insist on citations back to the source passage. Without them nobody can verify an answer, which makes the system unusable for real work.
• Measure how often it correctly says it does not know. A system that always answers is a system that sometimes invents.
• Where your documents sit during processing is a decision made at build time and hard to reverse later.
• Fine-tuning is not an alternative to this for factual questions over changing documents.

What it is actually for

The honest use case is narrow and valuable: you have a large body of your own material, people need answers out of it, and finding those answers currently means somebody who knows where things are going to look.

Concretely: a firm's accumulated engagement letters and precedents, an insurer's policy wordings, a manufacturer's equipment manuals, years of technical reports, a knowledge base support staff currently search by guesswork. The work being replaced is not thinking, it is hunting.

What it is not for is anything where the answer needs to be defensible in a way a person will be held to. Not because it cannot produce good answers, but because the value has to be verifiable, which is why the citation question below matters more than any benchmark.

Why retrieval is the whole product

This is the part that decides whether you got value, and it is where most of the engineering actually goes.

When somebody asks a question, something has to decide which passages from thousands of documents are relevant. That decision is the product. Hand the model the right three paragraphs and almost any competent model gives a good answer. Hand it the wrong three and you get a confident, fluent, well-written answer that is wrong, which is considerably worse than no answer because it reads as authoritative.

So when a vendor's pitch is mostly about which model they use, they are talking about the cheap, interchangeable part. The questions that matter are how documents get split up, how relevance is judged, what happens when a question spans several documents, and how the system behaves when the answer genuinely is not in the material.

That last one is the quiet killer. A system that always produces an answer will invent one when your documents do not contain it. The behaviour you want is for it to say it does not know, and to be right about that.

The four questions to put to any vendor

"Does every answer cite the passage it came from, and can I click through to it?" This should be non-negotiable. Without it there is no way for the person reading an answer to check it, which means either they trust it blindly or they go and look it up themselves, and the second one means you have bought nothing. With citations, the system becomes a very fast way of finding the right paragraph, which is genuinely useful even when the summary is imperfect.

"How often does it correctly decline to answer?" Ask for this as a number, tested on questions deliberately outside the document set. A vendor who has not measured it has not thought about the main failure mode.

"Where do the documents live while this runs, and who can reach them?" For anything client-confidential this usually decides the architecture. Running inside your own cloud account means the material never sits in a vendor's environment and you keep the system if the relationship ends. It also means somebody has to own that infrastructure, which is a real ongoing job rather than a footnote.

"What happens when a document is updated or withdrawn?" Document sets change. A superseded policy wording or a withdrawn precedent must stop being retrievable, and that requires a deliberate process rather than a one-off load at setup. Ask how it works and who runs it.

What it costs, and the part that catches people

Two costs, and the second one is where the surprises live.

The build is a project. Loading and splitting the documents, getting the retrieval quality to a usable standard on real questions, wiring the citations, building the place where people actually ask, and defining how updates flow in. For a contained document set that is weeks rather than months, and most of the time goes into retrieval quality rather than anything else.

The running cost is per question, and it scales with use rather than with the size of your document set. That means it behaves the opposite way to how people expect: loading ten times more documents costs little extra, but ten times more people asking costs roughly ten times more. Ask for cost per question at your expected volume, not a platform fee.

Then size the value honestly. Count how many times a week somebody currently hunts for an answer, how long that hunt takes, and what their time costs. Forty hunts a week at twelve minutes each is eight hours a week, so somewhere around four hundred hours a year. That is the number a quote has to beat, and it is usually a clearer case than any productivity claim in a deck.

Where this sits against the alternatives

Against fine-tuning. Fine-tuning adjusts how a model writes and reasons in a domain. It is not a way to load facts you need to keep current, and a fine-tuned model still cannot tell you which document an answer came from. For factual questions over documents that change, retrieval is the right tool and fine-tuning is an expensive detour.

Against just buying search. If your people mostly need to find the document rather than the answer, good search is cheaper and more predictable. The case for this only gets strong when the answer lives scattered across several documents and assembling it is the work.

Against doing nothing. Genuinely the right answer when the document set is small enough that people already know where things are. The value here scales with how much material there is and how often the person who knows is unavailable.

One framing to keep straight with your team: this sits alongside the people doing the work. It finds and drafts, they judge and own the answer. Deployed as a replacement for expertise it gets distrusted quickly and then ignored, which is the most common quiet failure of these systems.

FAQ

Is RAG as a service better than building it ourselves?

It depends on whether you have someone who will own it after launch, which is the question that actually matters. The build is well-understood enough that a capable internal team can do it, but retrieval quality needs iteration against real questions, and document updates need an ongoing process. Buying it as a service is mostly buying that ongoing ownership rather than the initial build.

Can it work over our documents if they are scans rather than files?

Yes, but that is a separate layer and a real cost, because a scanned page contains no text until something reads it. Expect the project to include a reading and validation step in front of the retrieval, and expect accuracy on poor-quality scans to need a human review path. Firms often discover their archive is photographs rather than documents only after signing.

How do we stop it giving confidently wrong answers?

Three things, and all three are design decisions rather than model choices. Require citations on every answer so a person can verify in one click. Set it to decline when retrieval confidence is low, and measure how often that judgement is right. And keep it pointed at answering rather than deciding, so a person stays between the output and any consequence.

How long before people actually use it?

Assume the first version will answer well on the questions you tested and poorly on a chunk of the real ones, because real questions are phrased worse than test questions. Budget a month of live use with someone watching the failures and adjusting retrieval. Systems that get abandoned are usually the ones launched without that period.

Key takeaways

• In a retrieval system, the step that selects relevant passages determines quality far more than the model does.
• Every answer should cite the passage it came from, otherwise nobody can verify it and the system saves no time.
• A system that never declines to answer will invent answers when your documents do not contain them.
• Running cost scales with how many questions are asked, not with how many documents you load.
• Fine-tuning is not a substitute for retrieval when facts change or citations are needed.

If you have a document set in mind, the fastest way to judge this is to test retrieval on twenty real questions people actually ask and count how many come back with the right passage.

Book a free strategy call

Related reading: AI agents for business · what is intelligent automation · do you need an AI automation consultant

Services: private knowledge systems · secure AI deployment