The model cannot answer what it never sees
A lot of teams start RAG by arguing about models. Bigger context window. Better embedding model. New reranker. Useful, sure. But the first failure is usually simpler: the answer was never retrieved. Once that happens, the model is decorating an empty plate.
The most common RAG bug looks boring in the logs.
Hit the button and watch the hidden AI surface area light up.
Chunking is product design
A chunk is not just a blob of text. It is a unit of meaning. If you split a policy table away from its heading, retrieval can find the table and lose the question it answers. If you make chunks huge, every match becomes noisy. Good RAG starts with boring document hygiene: titles, hierarchy, dates, owners, and clean boundaries.
Metadata is part of the answer. Treat it like source code, not decoration.
Measure retrieval before generation
Before you judge answer quality, run a tiny harness that asks: did the retriever find the document a human would use? That gives you a clean signal. If retrieval failed, fix search. If retrieval passed and the answer is bad, fix prompting, context shape, or the model.
Flip the switches and watch the story turn into a tiny operating model.
The best RAG systems are edited weekly
Your corpus changes. User language changes. Product names drift. So the system needs a weekly rhythm: inspect misses, merge duplicate docs, add synonyms, retire stale chunks, and replay the eval set. The glamorous part is the chat box. The durable part is search ops.