To deploy an LLM app to production, you wrap the model call in a small web service, put guardrails around it (timeouts, retries, rate limits, input validation), cache what can be reused, and add logging and tracing so you can see quality, latency, and cost for every request. The model is usually the easy part. The work is in making the service predictable when real users, real traffic, and real failures show up.

Why it matters: a prototype in a notebook hides problems that only appear under load. Provider APIs time out, responses arrive slowly, token bills grow faster than usage, and a prompt change can quietly break answers. This guide walks through a practical path from demo to a service you can trust, using FastAPI and Redis as the example stack.

What changes between a demo and production

• Concurrency: many users call the endpoint at once, so blocking code becomes a bottleneck.

• Failure: the model provider can be slow, rate-limited, or down, and your app must degrade gracefully.

• Cost: every token is billed, so unbounded prompts and repeated calls add up.

• Quality drift: prompt, model, or data changes can lower answer quality without throwing an error.

• Security: user input reaches a model that may call tools, which opens the door to prompt injection.

Step 1: Wrap the model in an API service

FastAPI is a common choice because it supports async endpoints, validates requests with Pydantic models, and generates API docs automatically. Define a request schema, call the model asynchronously, and return a typed response:

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()

class Ask(BaseModel):
    question: str = Field(max_length=4000)

@app.post("/ask")
async def ask(body: Ask):
    answer = await call_model(body.question)  # async client
    return {"answer": answer}

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()

class Ask(BaseModel):
    question: str = Field(max_length=4000)

@app.post("/ask")
async def ask(body: Ask):
    answer = await call_model(body.question)  # async client
    return {"answer": answer}

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()

class Ask(BaseModel):
    question: str = Field(max_length=4000)

@app.post("/ask")
async def ask(body: Ask):
    answer = await call_model(body.question)  # async client
    return {"answer": answer}

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI()

class Ask(BaseModel):
    question: str = Field(max_length=4000)

@app.post("/ask")
async def ask(body: Ask):
    answer = await call_model(body.question)  # async client
    return {"answer": answer}

Validation is your first guardrail: a maximum input length caps cost and rejects obviously abusive payloads before they reach the model. Keep the provider behind a small wrapper function so you can switch models or providers without touching your endpoints.

Step 2: Stream responses

Long answers can take several seconds to finish. Streaming tokens as they arrive makes the app feel faster even when total time is the same. FastAPI's StreamingResponse can forward chunks from the provider's streaming API to the client. Measure time to first token separately from total latency; users notice the first much more.

Step 3: Add timeouts, retries, and fallbacks

• Timeouts: set an explicit limit on every provider call. A hung request should fail fast, not tie up a worker.

• Retries with backoff: retry transient errors such as rate limits or brief outages, with exponential backoff and a small maximum number of attempts.

• Fallbacks: decide in advance what happens when retries fail: a smaller model, a cached answer, or a clear error message.

Do not retry requests that may have side effects, such as an agent sending an email, unless the operation is idempotent.

Step 4: Cache what you can

Two kinds of caching cut cost and latency:

1. Response caching: store answers in a fast key-value store such as Redis, keyed on a hash of the model, parameters, and normalized prompt. This works well for repeated questions like FAQs. Set an expiry so stale answers age out.

2. Prompt caching: several providers can reuse a long, stable prompt prefix, such as system instructions or reference documents, at a lower price. Put the stable part first and the variable part last. Learn more in what prompt caching is.

Step 5: Control cost and abuse

Put limits in code, not in a spreadsheet you check once a month:

• Rate-limit requests per user or API key.

• Cap max_tokens for outputs and trim conversation history to a budget.

• Log input and output tokens on every request and alert when daily spend crosses a threshold.

• Route simple tasks to smaller, cheaper models and reserve large models for hard ones.

If token pricing is unfamiliar, read how tokens drive API costs.

Step 6: Protect the app from bad input

Treat everything a user sends, and everything a tool or document returns, as untrusted. Keep secrets in environment variables or a secret manager, never in prompts. Limit which tools the model can call and require confirmation for actions that change data. The OWASP Top 10 for LLM Applications is a useful checklist of risks, starting with prompt injection, which is explained in why AI agents are vulnerable to prompt injection.

Step 7: Monitor latency, cost, and quality

Classic monitoring tells you whether the service is up. LLM apps also need to answer whether the answers are still good. Track at least:

• Latency: time to first token and total time, at the median and at high percentiles.

• Errors: provider errors, timeouts, and fallback rates.

• Cost: tokens and spend per request, per user, and per feature.

• Quality: user feedback, automated evaluation scores on sampled traffic, and refusal or empty-answer rates.

Tracing tools built on OpenTelemetry, or LLM-specific platforms, let you follow one request through retrieval, model calls, and tools. The difference between uptime monitoring and this deeper view is covered in AI observability vs monitoring.

Step 8: Ship changes safely

Prompts and model versions are code. Keep them in version control, run an evaluation set in CI before each release, and roll changes out gradually so you can compare quality and cost before everyone gets the new version. Package the service in a container so the same build runs locally, in staging, and in production. The broader practice of operating LLM systems this way is often called LLMOps; see what LLMOps is.

Pre-launch checklist

• Input validation and maximum lengths on every endpoint

• Timeouts, retries with backoff, and a defined fallback

• Streaming for long responses

• Response cache with expiry, and a stable prompt prefix

• Rate limits, output token caps, and spend alerts

• Secrets outside prompts and code; restricted tool permissions

• Tracing with latency, tokens, and cost per request

• An evaluation set that runs before every prompt or model change

FAQ

Do I need Kubernetes to deploy an LLM app?

No. If you call a hosted model API, a container on a managed platform is often enough at the start. Self-hosting open models adds GPU serving concerns that are a separate project.

Is FastAPI required?

No. Any web framework works. FastAPI is popular for LLM services because of async support, request validation, and automatic docs.

How do I know if quality dropped after a change?

Keep a fixed set of test questions with expected behavior, score new versions against it before release, and sample live traffic for ongoing review.

Learn to build and ship LLM systems

The AI Engineering Course is an eight-week live program for developers who already work with Python. Its curriculum runs from LLM application foundations and tool use to retrieval, evaluation and observability, serving and deployment, ending with a capstone you build, break, evaluate, and improve, using a stack that includes FastAPI and Redis. If you are moving into analysis instead, the data track covers SQL, Python, and statistics.

• AI Engineering Course

• Data Analytics Course