How AI Engineers Turn Business Needs into AI Requirements and KPIs

Turning a business need into AI requirements means translating a vague request ("we want AI for support") into a written spec: the user problem, user stories, acceptance criteria, an evaluation set and the KPIs that will prove the system worked. It is the first thing forward deployed and applied AI engineers do on a client project, and it decides whether the final system gets adopted or quietly abandoned. A model that scores well on a benchmark but does not move a business metric is a failed project.

This guide walks through a practical discovery-to-spec workflow, the three layers of metrics every AI project needs, and a worked example you can adapt.

Why AI projects need a different kind of requirements

Traditional software requirements describe deterministic behavior: given input X, return Y. LLM applications are probabilistic. The same input can produce different outputs, quality is a distribution rather than a pass/fail, and failure modes include hallucinations, unsafe content and silent drift. That is why AI requirements must include how quality will be measured, not only what the system should do. Google's People + AI Guidebook makes the same point: define what success looks like for users before choosing a model.

This translation work is a defining skill of the role described in our guide to what a forward deployed engineer is.

Step 1: Frame the problem, not the solution

Stakeholders usually arrive with a solution ("build us a chatbot"). Your job in discovery is to find the underlying problem. Useful questions:

• What task takes too long, costs too much or produces too many errors today?

• Who does that task, how often, and what does a good outcome look like?

• How is the task measured now, if at all?

• What happens when the task is done wrong? Who is affected?

• What data, systems and permissions are involved?

End discovery with a one-sentence problem statement, for example: "Support agents spend about a third of their time routing tickets to the right queue, and misrouted tickets add a day to resolution."

Step 2: Write user stories and acceptance criteria

User stories keep the spec anchored to real people. The classic format is "As a [user], I want [capability] so that [outcome]", as explained in Atlassian's guide to user stories. For AI systems, add acceptance criteria that describe quality and boundaries:

• Functional: "The assistant suggests a queue and a priority for every new ticket."

• Quality: "Suggested queue matches the human label on at least 90% of a held-out test set."

• Boundaries: "If confidence is low, the ticket goes to manual triage instead of a guess."

• Operational: "Suggestions appear in under three seconds and cost less than a set amount per ticket."

Step 3: Define three layers of metrics

Mixing these layers is the most common mistake in AI projects. Keep them separate and connected.

Business KPIs

The outcome the sponsor cares about: average resolution time, cost per ticket, conversion rate, revenue per rep. These move slowly and are influenced by many factors, so agree on a baseline before launch.

Product and adoption metrics

Signals that people actually use the system: share of tickets where the suggestion was accepted, how often agents override it, time saved per task. If adoption is low, business KPIs will not move.

Model and system metrics

What the engineering team measures directly: accuracy against labeled examples, groundedness for retrieval systems, refusal and error rates, latency and cost per request. Our explainer on AI evals covers how teams build these test sets.

The spec should state how each layer connects: "If routing accuracy reaches 90% and agents accept at least 70% of suggestions, we expect resolution time to drop measurably within one quarter."

Step 4: Build the evaluation set early

Before writing much code, collect real examples with the correct answer: 100 to 300 past tickets labeled by experienced agents is a good start. Include edge cases, ambiguous inputs and examples where the right answer is "send to a human". This set becomes your acceptance test, your regression suite and the evidence you show stakeholders. Provider documentation such as the OpenAI evals guide shows how to run these checks systematically.

Step 5: Capture constraints and risks

• Data: where it lives, who owns it, what is sensitive and what cannot leave the client's environment.

• Integration: which systems the AI must read from or write to, and with which permissions.

• Risk: what harm a wrong answer could cause and which controls are needed. The NIST AI Risk Management Framework offers a shared vocabulary for this conversation.

• Budget: acceptable latency and cost per task, which shape model choice.

A one-page AI spec template

1. Problem statement and current baseline

2. Users and user stories

3. Acceptance criteria: functional, quality, boundaries, operational

4. Metrics: business KPI, adoption metric, model metrics, with targets

5. Evaluation set: size, source, who labeled it

6. Constraints: data, integrations, security, budget

7. Out of scope for the first version

8. Launch plan: pilot group, review date, rollback criteria

Common mistakes

• Starting with a model or framework before defining success.

• Using only model metrics and never checking whether the business KPI moved.

• Skipping the baseline, so nobody can prove improvement.

• No fallback path when the system is uncertain.

• Writing the spec alone instead of reviewing it with the people who do the work.

FAQ

Who writes AI requirements: product managers or engineers?

Both. On client-facing AI projects, forward deployed and applied AI engineers often lead discovery because they understand what is technically feasible and how to measure it.

How precise should KPI targets be?

Precise enough to decide go or no-go. Use the baseline to set realistic targets and revisit them after the pilot.

Do I need labeled data before building?

You need at least a small, trustworthy evaluation set. Without it you cannot tell whether a prompt or model change made things better or worse.

Learn the full workflow, from discovery to handoff

Translating business needs into measurable AI systems is a skill you build by doing it on real projects. If you want guided, live practice, explore these programs:

• AI Engineering Course

• Data Analytics Course