Your AI Provider Is Down. Does Your “Fallback Model” Quietly Break the Product?
Build tested provider fallbacks with compatible behavior, privacy checks, circuit breakers, clear degradation, and fast rollback.
Availability is not the only property that can fail over
Teams add a second model provider to survive outages or rate limits. During an incident, requests move to the fallback and the service appears online. But the alternate may format output differently, refuse a task, call tools in another pattern, or process data under different terms. A green status page can hide a degraded product.
Failover is a product behavior. It changes the system that answers the user. The fallback should be tested against the same task, safety, privacy, and latency requirements as the primary. Otherwise the switch exchanges one outage for an unknown quality incident.
Start with a clear failure policy. Decide which errors trigger fallback, which return a retryable message, and which must stop because the alternate provider is not approved for that data class.
Define compatibility at the task level
Provider APIs may differ in schemas, tool-call formats, context limits, and streaming behavior. Build adapters that normalize only what your application truly needs. Do not erase differences that affect meaning. Validate structured outputs locally and convert provider-specific failures into stable internal error categories.
Run both providers on a representative evaluation set. Compare completion quality, refusal behavior, tool-call accuracy, citation support, content safety, latency, and total cost. Include cases where the correct outcome is to ask or abstain. Evaluate the actual prompts and tools, not a simplified text-only benchmark.
Check privacy and contractual eligibility for each route. A fallback that sends confidential data to an unapproved provider is not a valid reliability measure. Store data-class rules alongside routing policy and fail closed when the provider is not permitted.
Use circuit breakers and controlled recovery
Trigger failover on defined signals such as repeated connection errors, provider timeouts, or rate-limit responses. Use bounded retries with backoff before switching, and prevent retry storms. A circuit breaker should open for a defined period, then allow controlled probes to test recovery.
Avoid sending duplicate side-effecting requests to both providers. If a tool action may already have executed, check its status before retrying. Use idempotency keys on mutations. Keep model generation separate from execution so changing providers cannot duplicate an external action.
Expose the degraded state when it affects the user. If the alternate can answer only a narrower subset, state that limitation or offer a manual path. Do not present a lower-quality result as equivalent simply because it arrived faster.
Test the failure itself
Use fault injection in staging: timeout, malformed response, partial stream, rate limit, unavailable region, tool result delay, and recovery while requests are queued. Verify that the circuit opens, fallback receives eligible traffic, and normal routing resumes without duplicating work. Test provider-specific output differences in the UI.
Monitor primary and fallback separately. Track fallback rate, success by task class, correction rate, user complaints, latency, and cost. Alert if traffic remains on fallback after the primary recovers or if a task category starts failing disproportionately.
Keep model and prompt versions explicit in traces. After a provider changes its model, rerun the failover set. A fallback configuration that worked last quarter may no longer behave the same after a silent or announced update.
Practice switching back
Run periodic failover drills with synthetic or approved data. Confirm the team can disable a provider, switch routes, and return safely. Document who can change routing and how the change is reviewed. Keep a kill switch for tools with external side effects if uncertainty rises during an incident.
Multi-provider support can reduce dependency risk, but only if the alternate is understood. Test comparable tasks, preserve privacy rules, make degradation visible, and verify recovery. Resilience means maintaining a trustworthy outcome under failure, not merely keeping an endpoint reachable.
Include the customer-support team in failover drills. They need to recognize the degraded mode, explain what users can still do, and route reports to the right owner. If the fallback changes output quality or available features, prepare a short status message that states the limitation without exposing internal incident details. Communication is part of continuity.
After each drill, capture the time from failure detection to safe routing and the time needed to restore the primary. Review any manual steps that depended on one engineer's memory. The result should be a small runbook with exact dashboards, escalation contacts, and rollback commands, tested by someone other than its author.