
Dan Patiño
AI Strategy & Innovation at Coderhouse
Artificial Intelligence
An AI system runbook is the operating manual that lets a client's team run, monitor and fix an LLM application after the engineers who built it step away. It goes beyond a classic operations runbook: besides servers and alerts, it documents prompts, model versions, retrieval sources, evaluation thresholds, costs and what to do when answer quality quietly degrades. A good handoff is the difference between an AI project that keeps delivering value and one that gets switched off the first time something goes wrong.
This step-by-step guide shows how applied and forward deployed AI engineers prepare a runbook and hand off a production AI system to a client organization.
Why AI systems need a different kind of runbook
Traditional software usually fails loudly: an error, a crash, a timeout. AI systems also fail quietly. The service is up, latency looks normal, but answers are less accurate because a document source changed, a model version was updated or users started asking questions nobody tested. The Google SRE book remains the reference for operational practices such as monitoring, on-call and postmortems; an AI runbook adds the layer that covers quality, not only availability. Frameworks such as the NIST AI Risk Management Framework also stress ongoing monitoring and clear ownership of AI risks after deployment.
Step 1: Document the system the client is inheriting
Purpose and scope: the user stories, what the system should and should not do, and the KPIs agreed during discovery.
Architecture diagram: inputs, retrieval, prompt assembly, model calls, tools and outputs, plus every external system it touches.
Data flows and governance: what data enters the system, where it is stored, retention rules, and how personal data is masked in logs.
Dependencies: model providers, vector database, identity provider, APIs and their rate limits.
Owners: a named person or team for the application, the data sources, the model budget and security.
Step 2: Version everything that changes behavior
In an LLM application, behavior depends on more than code. The runbook should state where each of these lives and how to change it safely:
Prompt templates and system instructions, versioned like code.
Model names and parameters, with a pinned version where the provider allows it.
Retrieval configuration: chunking, embedding model, index refresh schedule and reranking settings.
Tool definitions and the permissions each tool uses.
Include a rule that any change to these goes through the evaluation suite before release.
Step 3: Hand over the evaluation suite and quality thresholds
The client needs a way to answer "is it still working well?" without the original team. Hand over:
A test set of representative questions with expected answers or grading criteria, including past failures.
Automated checks: format validity, groundedness in retrieved sources, refusal behavior and safety tests.
Thresholds that define a release as acceptable, for example a minimum pass rate on the test set.
Instructions to run the suite on demand and on a schedule.
Step 4: Define monitoring and alerts
Specify which dashboards to watch and which alerts page someone. Useful signals include error rate, latency, token usage and cost per request, tool failure rates, user feedback such as thumbs-down rates, and sampled quality scores on live traffic. For each alert, the runbook should say who receives it and what the first action is. The SRE workbook chapter on on-call is a practical reference for keeping alerts actionable instead of noisy.
Step 5: Write incident playbooks for AI-specific failures
Each playbook should be short: symptoms, how to confirm, immediate mitigation, and who to escalate to.
Quality regression
Symptoms: more complaints, lower eval scores. Check recent changes to prompts, models or data sources first, then compare traces of failing requests against passing ones. Our step-by-step guide to debugging production LLM applications walks through that diagnosis.
Model provider outage or slowdown
Document the fallback: a secondary model, cached answers for common questions, or a graceful message that routes users to a human.
Cost spike
Identify which feature or user group drove the increase, then apply limits such as maximum tokens, rate limits or caching.
Security or prompt injection incident
Steps to disable risky tools, revoke tokens, preserve logs and notify the security owner.
Stale or wrong knowledge
How to re-index documents, remove a bad source and verify that retrieval returns the corrected content.
Step 6: Plan rollbacks and safe changes
Explain how to roll back to the previous prompt, model or index version in minutes, and how to use feature flags or gradual rollouts for risky changes. A rollback that requires the original developer is not a rollback the client can use.
Step 7: Run the handoff, not just the document
Walkthrough: present the architecture, runbook and dashboards to the client's engineers and the business owner.
Shadowing: the client team watches while your team handles real alerts and changes.
Reverse shadowing: the client team operates the system while your team observes and corrects.
Game day: simulate an incident, such as a provider outage or a quality drop, and let the client resolve it with the runbook.
Sign-off: agree on explicit acceptance criteria, open issues and a support window.
A simple runbook template
System overview and KPIs
Architecture and data flows
Owners and escalation contacts
Configuration and versioning (prompts, models, retrieval, tools)
Evaluation suite and release thresholds
Monitoring, dashboards and alerts
Incident playbooks
Rollback and change procedures
Known limitations and open risks
Why this skill matters for your career
Client-facing AI roles are judged by what keeps working after launch. Engineers who can explain a system to technical and non-technical stakeholders, and leave behind documentation a client can actually use, build trust quickly. If that path interests you, our explainer on what a forward deployed engineer does and how to become one covers the role in depth.
FAQ
How long should an AI runbook be?
Long enough to cover every alert and common incident, short enough to use under pressure. Many teams keep a concise main document with links to deeper references.
Who should own the runbook after handoff?
The client team that operates the system, with a named owner responsible for keeping it updated after every significant change.
Is a runbook needed for an internal prototype?
A lightweight version helps even then, but it becomes essential once real users and business processes depend on the system.
Keep learning with Coderhouse
Our AI Engineering Course (Forward Deployed AI Engineering) runs for 8 weeks with one 2-hour live class per week and about 6 to 8 hours of work weekly. It is designed for developers comfortable with Python and covers client discovery, RAG, evaluation and observability, agents and enterprise integration, reliability, governance and deployment, ending with a capstone you build, evaluate and hand off with runbooks. To strengthen how you measure quality and impact, the Data Analytics Course covers SQL, Python, statistics and experimentation.
I'm Dan Patiño, head of AI Strategy & Innovation at Coderhouse. My day-to-day work involves merging the tactical management of e-commerce (CRO, Email Marketing and SEO) with the development of disruptive solutions. I specialize in building internal AI-powered apps to automate tasks and boost innovation within the team. I firmly believe that technology is strategy's best ally. To dive deeper into my professional journey, I'll be waiting for you on my LinkedIn profile.