AI SLOs for Production: Measure Reliability When Outputs Are Probabilistic

Traditional service-level objectives are comfortable because a request either returns within a latency target or it does not. AI systems complicate the picture. A response can arrive quickly and be wrong, a tool call can be valid but unauthorized, or an agent can complete a workflow while quietly spending ten times the expected budget. As agentic products move from demos into customer operations, teams need a reliability language that includes behavior without pretending outputs are deterministic. An AI SLO is a measurable promise about a workflow under defined conditions. It is not a claim that a model is always correct. The promise might be that 99 percent of approved support answers cite a current source, that 99.9 percent of account-change attempts require an approved human gate, or that 95 percent of document extractions complete within two minutes without invalid fields. Making the promise explicit exposes where the product is ready for autonomy and where it still needs review. ## Define the unit of reliability Choose a user-visible unit such as a completed answer, a resolved ticket draft, a submitted report, or an agent run. Avoid using raw tokens or model calls as the primary unit because one user request may contain many calls. For each unit, define success, acceptable degradation, and a hard failure. A support answer that says it is uncertain and routes to a person may be a successful safe outcome; a confident answer from an obsolete policy is not. Write separate objectives for availability, latency, quality, safety, cost, and recovery. Availability asks whether the workflow can be started. Latency asks whether it finishes within a useful time. Quality asks whether the output meets task requirements. Safety asks whether prohibited actions and data exposures stay below a strict threshold. Cost asks whether the unit economics remain viable. Recovery asks how quickly traffic can move to a fallback after a provider or model incident. ## Use error budgets carefully An error budget is the amount of failure a team can tolerate during a period. For conventional availability, a few failed requests may be an acceptable budget. For high-impact safety events, the budget may be zero for the specific class of failure. Do not average an unauthorized payment attempt with a harmless formatting error and declare the combined rate acceptable. Weight failure classes by consequence and create hard stop rules for privacy, security, and irreversible side effects. Quality SLOs need sampling and labels. Combine deterministic validators with human review, user correction signals, and carefully calibrated model-assisted grading. Publish the sampling method and confidence limits so a green dashboard does not imply more certainty than the data supports. If only a small percentage of conversations are reviewed, say so. A low observed error rate in a biased sample is not proof of a low true error rate. ## Instrument the full run Emit a trace for the user request, prompt version, model, retrieved evidence, tool calls, approvals, retries, and final state. Attach dimensions that help diagnose changes without exposing sensitive content: tenant class, workflow, language, policy version, and outcome category. Record cost and latency per step. An agent can meet an end-to-end latency target while one slow tool consumes most of the budget; step-level data tells you where to improve. Create alerts that point to action. A quality alert should say which workflow and slice crossed its threshold and whether traffic can be reduced. A cost alert should identify loops, oversized context, or a routing change. A safety alert should activate the kill switch and preserve evidence. Avoid alerting on every model disagreement; alert when the disagreement changes a user-visible or consequential outcome. ## Design degraded modes in advance Every important AI workflow needs a behavior for provider outage, tool failure, missing context, evaluator uncertainty, and budget exhaustion. The degraded mode may be a search-only answer, a draft without sending, a static help center, or a human queue. Test it as part of the release rather than discovering it during an incident. Keep the switch outside the model so a model cannot decide whether it should be disabled. Review SLOs after product scope changes. Adding a new language, a write-capable connector, longer memory, or a new customer population can alter risk even when uptime is unchanged. Reliability is the ability to deliver a useful and bounded outcome repeatedly. Define the outcome, measure its evidence, reserve stricter budgets for serious harm, and make recovery routine before scale turns uncertainty into an outage.