Latency Budgets for RAG Applications

A RAG application has a latency budget whether you have written it down or not. If you haven’t, the budget is whatever the slowest component happens to do on a bad day, and the user experience follows.

The template I use is simple: allocate milliseconds across retrieval, rerank, prompt assembly, generation, and post-processing, then enforce each as a soft deadline with a clearly documented fallback.

The goal is not to hit every budget every time. The goal is to notice quickly when you don’t, and to have already decided what degrades gracefully and what is allowed to fail loudly.