General · · 4 min read
When RAG Actually Earns Its Complexity
RAG is sold as mandatory for enterprise AI, but it's not. Here's when it genuinely solves a problem and when you're just adding bloat.
Every enterprise AI vendor deck I see right now has a slide that says RAG, usually in a big font, usually with an arrow pointing at the word "trust." And I get it, it sells. But it's not mandatory, and I'd say treating it that way is one of the quicker routes to a bloated build that nobody asked for.
So let's be straight about when it earns its keep and when it doesn't. You already know what it is, so I'll skip that bit.
Where RAG genuinely earns its complexity
First, a domain-specific knowledge base that the model could never have seen. Internal policies, product documentation, a niche body of technical material. If the answer lives in your data and nowhere else, the model has nothing to draw on, and retrieval is the honest fix. Fine-tuning for that is slower, pricier and goes stale the moment the documents change.
Second, regulated industries where someone has to point at the source. Legal, healthcare, finance, anything where "the model said so" gets you laughed out of an audit. Here the citation is the product. RAG gives you a paper trail, which is the whole point.
Third, products where a hallucination is expensive. Not embarrassing, expensive. If a wrong answer means a bad decision, a compliance breach or a customer who never comes back, grounding the output in retrievable text is worth the plumbing. You're buying down risk, and that's a perfectly good reason to build something.
Where RAG is premature for a studio build
Pre-PMF is the big one. If you don't yet know what users actually do with the thing, you don't know what they need to retrieve. I've seen (well, I've watched the pattern play out, you know how it goes) teams spend weeks tuning chunking strategies for a product whose core workflow changed twice before launch. All that retrieval work, gone. You're optimising an answer to a question you haven't finished asking.
Then there's the case where your whole corpus fits in the context window and coverage is already fine. Just put the documents in the prompt. Seriously. It's simpler, it's easier to debug, and there's no retrieval layer to quietly return the wrong chunk and make you doubt everything. Complexity you don't need is just a place for bugs to live.
And cost-sensitive MVPs. A vector store and an embedding pipeline, plus a re-ranker and the monitoring to keep it all honest: that's real ongoing spend and real ongoing attention. If you're trying to find out whether anyone cares, that budget is better spent on talking to users. I'm not saying never. I'm saying not yet.
Four questions before you commit
Run these before anyone opens a vector database tab.
One: does the answer depend on information the model can't have? If it's general knowledge or reasoning, you probably don't need retrieval at all.
Two: does the user, or a regulator, need to see where the answer came from? If yes, that pushes hard towards RAG. If no, it's a nice-to-have, which is a different conversation.
Three: does the source material change often enough that baking it in or pasting it in stops working? Static and small, skip it. Large and shifting, lean in.
Four: can you describe, specifically, what a wrong answer costs? If you can't, you're guessing about the value of the whole architecture. Get a feel for the cost first, then decide how much machinery it justifies.
Two or more clear yeses and I'd start designing. Zero or one and I'd hold off, absolutely, and ship something simpler.
What trustworthy actually needs beyond RAG
Here's the bit that gets lost. RAG doesn't make a product trustworthy. It makes one failure mode less likely while quietly introducing others, like retrieving the wrong passage with total confidence. So the verification work sits on top, and honestly it matters more than the retrieval does.
You need evals, real ones, run against questions that resemble what users truly ask, and re-run every time something changes. You need logging good enough that when something goes wrong you can see what was retrieved, what the model did with it, and what the user saw. Without that you're debugging by vibes. And you need a human review loop, even a thin one, so someone is actually reading a sample of outputs and flagging the strange ones.
None of that is glamorous. All of it is what separates "we have RAG" from "we can show you it works."
Sequencing, then
Ship the plain version first: good prompts, the documents in context if they fit, evals and logging from day one. Watch where it breaks. If the breakage is "it doesn't know our stuff" or "people need the source," that's your signal, and retrieval goes in as a response to a real problem rather than a line on a slide. Build the review loop alongside it, not after. And if you want to learn the failure modes cheaply, point an agentic setup at your own internal data first, before a client pays for the lesson.
Anyway, that's the order. Evals, then logging, then retrieval when it's earned. The arrow on the vendor's slide can wait.
Filed under General