LLM Latency: Why AI Apps Feel Slow and How Engineers Speed Them Up

LLM latency is the time between a user sending a request to an AI application and getting a useful answer back. In practice it has two parts people feel: time to first token, how long the screen stays blank before anything appears, and total generation time, how long the model takes to finish writing. A slow AI app is rarely slow for one reason. Latency piles up across model choice, prompt size, output length, retrieval, tool calls, and the number of round trips your code makes.

Latency matters because users judge an AI feature by how it feels, not by how clever the architecture is. A support assistant that answers correctly in twelve seconds often loses to a simpler one that starts responding in one second. Latency also tracks closely with cost: most of the changes that make an LLM app faster also make it cheaper to run. This guide explains where the time goes and the practical steps AI engineers use to speed things up.

Where LLM latency comes from

Before optimizing anything, break a request into its steps. A typical retrieval-augmented chat request looks like this:

• Network and queueing: the request travels to the model provider and may wait for capacity.

• Pre-processing: rewriting the user query, classifying intent, or checking for safety issues, often with extra model calls.

• Retrieval: embedding the query, searching a vector database, and sometimes reranking results.

• Prompt processing: the model reads the system prompt, history, and retrieved context. Longer inputs take longer, but usually less than people expect.

• Generation: the model writes output tokens one at a time. This is almost always the biggest chunk.

• Tool calls and loops: agents may call tools, wait for the results, and call the model again several times.

OpenAI's latency optimization guide puts it bluntly: generating tokens is almost always the highest-latency step, and as a rough heuristic, cutting 50% of output tokens may cut about 50% of latency. Cutting 50% of the prompt, by contrast, may only improve latency by 1 to 5% unless you work with very large contexts.

Step 1: Measure before you optimize

Guessing where time goes is the most common mistake. Add tracing so every request records the duration of each step: retrieval, each model call, each tool call, and post-processing. Track at least three numbers:

• Time to first token (TTFT): how long until the first character reaches the user.

• Output tokens per second: how fast the model writes once it starts.

• End-to-end latency at p50 and p95: the median tells you the typical experience; the 95th percentile tells you how bad the slow requests get.

Set a latency budget per step, such as 300 ms for retrieval and 1 second to first token, so you know which part to fix first. Tools for AI observability make this visible without building dashboards from scratch.

Step 2: Generate fewer tokens

Because generation dominates, shorter outputs are the fastest win:

• Ask for concise answers and show short examples of the format you want.

• For structured outputs, use compact JSON with short field names and only the fields you actually use.

• Set a sensible maximum output length so a runaway response cannot take thirty seconds.

• Avoid asking the model to restate the question or explain its reasoning when users do not need it.

Step 3: Pick the right model for each step

Smaller models usually run faster and cost less. Many apps use one large model for everything, including simple jobs like classifying intent or rewriting a query. Route those narrow steps to a small, fast model and keep the large model for the final answer. Test quality on a fixed evaluation set before and after the switch so speed does not quietly cost you accuracy.

Step 4: Make fewer requests and parallelize the rest

Every model call adds a round trip. If two sequential steps can be done in one prompt, such as rewriting a query and deciding whether retrieval is needed, combine them and return both results as named JSON fields. When steps do not depend on each other, run them in parallel. For example, a safety check and the main answer can start at the same time, and you can discard the answer if the check fails.

Agents deserve special attention. An agent that calls the model five times in a loop will be slow no matter how fast each call is. Limit the number of steps, let the model call multiple tools in parallel when possible, and hard-code steps that do not need judgment.

Step 5: Reuse work with caching

• Prompt caching: providers can reuse the processed prefix of a prompt across requests. Keep static instructions at the start and put dynamic content, like retrieved documents and chat history, at the end. See how prompt caching works in OpenAI's docs.

• Response caching: store answers to frequent, identical questions and serve them instantly.

• Semantic caching: reuse answers for questions that mean the same thing, with a similarity threshold you test carefully to avoid wrong matches.

• Retrieval caching: cache embeddings and search results for popular queries.

Step 6: Make users wait less with streaming

Streaming sends tokens to the interface as they are generated, so users start reading within a second instead of staring at a spinner. Both OpenAI and Anthropic describe streaming as a core technique; Anthropic's guide to reducing latency recommends it alongside choosing the right model and keeping prompts and outputs tight. For multi-step agents, show progress, such as "searching documents" or "checking your order," so the wait feels purposeful.

Step 7: Do not default to an LLM

Some steps do not need a model at all. Confirmation messages, fixed menus, simple lookups, and calculations are faster and more reliable as regular code. Classic engineering still applies: database indexes, connection pooling, timeouts, and retries with backoff matter as much in AI apps as anywhere else.

Speed, quality, and cost trade-offs

The rule that keeps teams honest: every optimization gets checked against an evaluation set. A faster app that gives worse answers is not an improvement.

Frequently asked questions

What is a good latency for an LLM app?

It depends on the use case. Chat interfaces should usually show the first token within one to two seconds. Background jobs, like summarizing documents overnight, can take much longer without hurting anyone.

Does a longer prompt make responses slower?

Somewhat, but output length usually matters far more. Very large contexts, such as long documents or many retrieved chunks, are the exception, so prune retrieval results to what the model needs.

Is latency optimization a skill employers look for?

Yes. Production AI roles expect engineers to handle cost, latency, reliability, and evaluation, not just get a demo working.

Learn to ship fast, reliable AI apps

If you already write Python and want to build production AI systems, the AI Engineering Course is an 8-week live program with one 2-hour class per week that covers LLM APIs, retrieval, evaluation and observability, agents, and deployment with retries, timeouts, and fallbacks. If you want to analyze product and usage data, the Data Analytics Course teaches SQL, Python, statistics, and experimentation.

• AI Engineering Course

• Data Analytics Course