Personal · · 4 min read
RAG is a tool. Sometimes it's the wrong one.
RAG solves specific problems—stale knowledge bases, regulatory citations, costly errors—but it's often added prematurely. Here's when to use it, and when to shi
Every enterprise AI vendor I read at the moment is making RAG sound like a seatbelt. Mandatory, obviously, why would you even ask. But it isn't always, and I've been feeling that pressure build for a while now, so let's just say it plainly: RAG is a tool, and sometimes it's the wrong one for the job.
I've said this before, so I won't relitigate it. What I haven't done is give you the actual checklist, the one you can run on a Thursday afternoon before anyone commits engineering time to a vector database. So here it is.
When RAG earns its complexity
First, a domain-specific knowledge base that changes. If your product answers questions from a body of material the base model never saw (internal policies, product documentation, a niche technical corpus) and that material gets updated, RAG is right. Fine-tuning on it would go stale the moment someone edits a page. Retrieval keeps the answer tied to the current source.
Second, regulated industries where you have to show your working. If a user, an auditor or a regulator can ask "where did that come from?", you need a citation that points to a real document, not a confident paragraph. RAG gives you that trail almost for free, which is the whole point.
Third, products where a wrong answer is expensive. Think about a tool where a hallucinated figure or an invented clause lands in front of someone who then acts on it. Grounding the model in retrieved text won't make it perfect (it can still misread a good source) but it narrows the space where it can make things up.
If you're nodding at one of those three, good. Keep reading anyway, because the next bit is where studios waste the most money.
When RAG is premature
Pre-PMF products, first. You don't yet know what your users actually ask, so you can't know what your knowledge base needs to contain. Building a careful retrieval pipeline before you understand the questions is like tuning an engine for a road you haven't found yet. You'll build it, then rebuild it, then wonder why you spent a month on chunking strategy.
Second, small context windows with adequate coverage. If your whole corpus fits comfortably in the prompt, just put it in the prompt. I mean it. Modern context windows are generous, and stuffing the relevant material straight in is simpler, easier to debug and has one fewer thing to break. Retrieval adds a failure point: the model can only answer well if the right chunk came back, and sometimes it doesn't.
Third, cost-sensitive MVPs. RAG isn't free. You're paying for embeddings, storage, retrieval infrastructure and the engineering hours to keep it all behaving. If you're trying to find out whether anyone wants the thing, spend that money on finding out whether anyone wants the thing.
Four questions before you commit
Run these in order, and be honest, because nobody's marking you.
Does the answer depend on information the base model can't have? If no, stop here. You probably don't need retrieval.
Does that information change often enough that baking it in would go stale? If it's static and small, a long prompt does the job.
Will someone need to verify where an answer came from? If yes, that's your strongest argument for RAG, and it should carry real weight in the decision.
What does a wrong answer cost you? Not in the abstract. Actually sit with it. If the honest answer is "mild embarrassment," you've got room to ship something simpler first.
Two or more firm yeses and RAG is probably earning its place. One or none, and you're likely adding complexity because a vendor deck made it feel normal.
Trustworthy takes more than retrieval
Here's the part that gets skipped. RAG doesn't make an AI product trustworthy on its own. It makes one specific failure less likely. Plenty of RAG systems retrieve the wrong passage, or the right passage and then summarise it badly, and nobody notices because nobody's checking.
So you need evals, a set of real questions with known good answers that you run every time something changes. Without them you're guessing whether your last tweak helped or quietly broke something.
You need logging, too. Record what was asked, what was retrieved and what the model said. When a user complains, you want to open the trace and see exactly where it went sideways, not shrug.
And you need a human review loop, at least while the product is young. Someone reading a sample of real outputs every week will catch problems that no automated metric flags. It's unglamorous, absolutely, and it's also the thing that separates a product people trust from one that merely claims to be reliable.
What to ship first
So, sequencing. Ship the simplest version that could work: a good prompt, the relevant material in context, and logging from day one. Build your eval set from the real questions that come in. Only when you see retrieval would genuinely fix a failure you can point to (stale information, missing citations, a corpus that no longer fits) do you layer RAG in, and by then you'll know exactly what it needs to do. That's the whole trick, really. Earn the complexity, don't inherit it.
Filed under Personal