Writing · Adapt Progress Evolve

Personal

RAG is a Tool, Not a Badge: When to Build It and When to Skip

RAG solves specific problems—domain knowledge, auditability, expensive errors—but it's often added prematurely. Decide based on real constraints, not vendor pre

Filed 22 October 2026 · 4 min read

Every enterprise AI vendor I hear from is making RAG sound mandatory. Like you can't ship a serious product without a vector database bolted on the side. You can. Sometimes you absolutely should. But the pitch skips the bit where you ask whether you actually need it.

So here's the verdict up front: RAG is a tool for specific problems, not a badge of seriousness. Let's work out which problems.

When RAG earns its complexity

First, domain-specific knowledge bases. If your product has to answer from material the base model has never seen (internal documentation, proprietary research, a client's policy library), retrieval is the honest answer. Fine-tuning for that is slower and goes stale the moment the documents change. RAG lets the knowledge move without retraining anything.

Second, regulated industries where you have to show your working. If a user, an auditor or a regulator asks "where did that come from?", you need a source you can point at. Retrieval gives you that trail almost for free, because the passage that grounded the answer is right there. A model answering from its own weights can't do that, no matter how confident it sounds.

Third, products where a hallucination is expensive. Not embarrassing, expensive. Think of a tool that tells someone what a contract clause means, or what a dosage guideline says. Grounding the answer in retrieved text won't make the model infallible, but it narrows the room for invention, and that's worth the plumbing.

When it's premature

Now the other side, which vendors rarely mention.

Pre-PMF products first. If you don't yet know what your users actually ask for, you can't design a retrieval layer for it. You'll build chunking strategies and embedding pipelines around guesses, then rip them out when the real usage shows up. I mean, it's a lot of engineering to get wrong before you've learned anything.

Then there's the small-corpus case. If everything the model needs fits comfortably in the context window, just put it in the context window. Seriously. A well-organised prompt with the full material pasted in is simpler, easier to debug, and has one fewer system that can quietly fail. Retrieval adds a step where the wrong chunk gets fetched and nobody notices.

And cost-sensitive MVPs. RAG isn't free: you're paying for embeddings, storage, a retrieval service, and the engineering time to tune it all. If you're trying to prove demand on a tight budget, that spend is better pointed at getting in front of users.

Four questions before you commit

Run these before anyone opens an architecture diagram.

Does the knowledge the product needs change faster than you'd want to retrain? If it's stable, you may not need retrieval at all.

Could the whole body of source material fit in a single prompt today? If yes, start there and revisit when it stops being true.

Does anyone downstream need to see where an answer came from? If the answer is a firm yes, retrieval moves from nice-to-have to required.

What happens when the product is wrong, and who pays for it? The higher that cost, the more the extra machinery is justified.

If you get "no, yes, no, not much", skip RAG for now. If you get the opposite, build it. The mixed cases are where judgement comes in, and that's fine, you just want to be making that call on purpose rather than because a slide deck told you to.

What trustworthy actually needs

Here's the bit that bugs me. RAG gets treated as the trust layer, as though retrieving documents makes an answer verifiable. It doesn't. It makes it grounded, which is a different thing. A model can retrieve the right passage and still misread it, or pick the wrong passage and sound perfectly sure.

Real trust comes from the boring stuff around the model. Evals, so you know how often it's right on the questions that matter, measured before and after every change. Logging, so when something goes wrong you can see what was retrieved, what was asked, and what came back. And human review loops, so a person catches the failures the evals didn't anticipate and feeds them back in.

Honestly, a product with no retrieval but solid evals and review is more trustworthy than one with a gorgeous retrieval stack and no way of knowing when it's failing. Retrieval is one input. The verification is the product.

What to ship first

So, sequencing. Ship the simplest thing that can talk to real users: a plain prompt, your best material in context, and logging switched on from day one. Build a small eval set out of what people actually ask. Get a human looking at outputs early. Then, once real usage shows you the corpus has outgrown the context window, or someone needs sources, or the cost of a wrong answer has climbed, layer retrieval in, and by then you'll know exactly what it needs to retrieve. That's the whole trick, really. Earn the complexity, don't pre-pay for it.