Personal · · 4 min read
RAG Is Not a Badge of Honor
RAG solves specific problems, not all of them. Here's how to know if your product actually needs it.
Every enterprise AI vendor I hear from lately is making RAG sound like the seatbelt of the industry. Skip it and you're basically reckless. Right?
Not really. I mean, I've been turning this over for a while, and the honest answer is that RAG is a tool for specific problems, not a badge of seriousness. Plenty of products don't need it. Some are actively worse off for having it. Anyway, here's how I'd think about it if I were sitting next to you with a whiteboard and a slightly cold coffee.
Where RAG earns its keep
First, domain-specific knowledge bases. If your product has to answer from a body of material the model never saw in training, say a company's internal policies, a niche technical manual, a constantly changing set of documents, then retrieval is doing real work. The model can't guess its way to your private knowledge. You have to hand it over.
Second, regulated industries where someone will ask "where did that answer come from?" and you need a real reply. Source attribution isn't a nice extra there. It's the product. A citation that points back to an actual passage is the difference between something a compliance person will tolerate and something they'll quietly kill.
Third, products where a hallucination is expensive. If a wrong answer costs a user money, trust, or a legal headache, grounding the output in retrieved text shrinks the space where the model can make things up. It doesn't remove it (more on that below), but it narrows it.
Where it's premature
Now the other side, the one the vendors skip. If you're pre-PMF, you don't yet know what your users actually ask. Building a retrieval pipeline before you know the questions is like paving a road before anyone's decided where the town goes. You'll pour effort into chunking and indexing for a use case that changes in a fortnight.
Then there's the small-corpus situation. If your whole knowledge set fits comfortably inside a modern context window and the coverage is already adequate, just put it in the prompt. Seriously. Fewer moving parts, fewer places to break. Building your own pipeline means owning connectors, permissions models, chunking, evaluation, and every failure mode that comes with them, and that's a lot of weight for something a long prompt might handle.
And cost-sensitive MVPs. Retrieval adds infrastructure, latency, and a pile of tuning work. If you're trying to learn fast on a thin budget, that complexity is a tax on exactly the thing you need most, which is speed.
Four questions before you commit
Here's the matrix, except it's not really a matrix, it's just four questions you answer honestly.
Does the model need knowledge it can't possibly have? If the answer lives in public, well-known territory, retrieval may add nothing.
Will someone demand to see the source? If yes, you're leaning towards RAG. If no, you've got room to wait.
What does a wrong answer cost? Low cost means you can tolerate a simpler build and learn from the misses. High cost pushes you towards grounding.
Do you already know the shape of your users' questions? If you don't, you're guessing, and guessing at architecture is expensive.
If you get a clear "yes" on the first two and a high cost on the third, build it. Mixed answers? Probably not yet. I'd rather you ship something small and learn than architect something grand and be wrong.
What trustworthy actually takes
Here's the bit I keep banging on about. RAG alone doesn't make a product trustworthy. People treat it like a finish line when it's barely the start line.
You need evals, for a start. Real ones, run regularly, that tell you whether the system is getting better or quietly rotting. Without them you're feeling your way through the dark and calling it confidence.
You need logging, so that when something goes wrong you can see what was retrieved, what the model did with it, and where it drifted. A pipeline you can't inspect is a pipeline you can't fix.
And you need humans in the loop, at least for the cases that matter. Review queues, spot checks, an easy way for users to flag a bad answer. It's unglamorous. It's also what actually makes the thing verifiable, because retrieval can still pull the wrong passage and the model can still misread the right one.
What to ship first
So, sequencing. Ship the simplest version that tests whether anyone cares. Stuff the context window, write your evals early, log everything, and get a human looking at outputs. Watch where it fails. If the failures cluster around missing knowledge or demands for sources, that's your signal, and that's when you layer retrieval in, with a clear reason and a way to measure whether it helped. Practising agentic RAG on your own internal data first is a cheap way to learn its failure modes before a client pays for them.
Funny thing is, the products that end up needing RAG usually tell you so themselves. You just have to build the small thing first and listen.
Filed under Personal