Every RAG demo looks good. Ten hand-picked questions, confident answers, a happy stakeholder in the room. Three weeks into production a prompt update breaks tool calling, retrieval starts returning stale chunks, and nobody notices until a customer does.

The uncomfortable fact is that retrieval quality and generation quality are two separate failure modes, and most teams only ever look at the second one because it’s the one you can see. Context precision and recall need their own dashboard, separate from answer correctness. Raise k and you improve recall but you also raise latency and cost, and nobody wants to be the one to say that trade-off out loud in the roadmap review.

My rule for any RAG rollout now: no production traffic until there is an evaluation suite that runs on every change, not just at launch. Lab evaluation validates feasibility. It says nothing about production, and production is the only environment that matters.