“Human in the Loop” Is Not a Safety Plan: Design AI Approvals That Catch Real Mistakes
Replace rubber-stamp confirmation with risk-based approvals bound to exact actions, evidence, identity, and a reliable audit trail.
A confirmation button can create false confidence
Many AI product designs add a human approval step and treat the risk as solved. In practice, reviewers may see a vague summary, approve quickly, and have no way to tell whether the underlying target or payload changes afterward. A human who lacks time, context, or authority is not an effective control. The design must help a person detect a consequential mistake.
This matters more as agents move from drafting toward taking actions across software tools. A model may prepare a refund, alter permissions, send an email, or modify production configuration. If the interface displays only “The agent wants to continue,” the reviewer is being asked to trust the system at exactly the moment when independent inspection matters most.
The goal is not to ask for approval constantly. Too many low-value prompts create fatigue. Instead, match review effort to impact, reversibility, uncertainty, and data sensitivity. Reading a public page can be automatic. Sending a message to a customer deserves a preview. Changing access to a sensitive workspace may require a second authorized reviewer.
Present an actionable review packet
An approval screen should state what will happen, which resource will change, who or what will be affected, and how to undo the action. Show the exact message, amount, permission, or configuration. Include concise source evidence and label facts the model inferred. Avoid presenting a confidence percentage without a calibrated meaning; reviewers tend to treat numeric precision as evidence.
Make the relevant difference visible. If the agent is updating an existing record, show old and new values. If it will send a message, show the recipient identity and content. If it will transfer data, name the destination and fields. Let the reviewer inspect the source system when the model's summary is insufficient.
Separate approval from execution. The user should be able to edit or reject the proposal. If an edit changes the target, amount, or content, regenerate the approval record. Bind the approval to a canonical representation of the action so that a later model step cannot alter it without a new confirmation.
Check the reviewer and the context
Verify that the approver has authority for this action. Being able to use the AI product should not automatically grant permission to approve every operation it proposes. For sensitive actions, use role-based policies, separation of duties, and a second reviewer. Re-check authorization at commit time because access may change while approval is pending.
Give the reviewer enough context to notice mismatches, but avoid dumping a full transcript. Highlight the source of each important field, the time it was retrieved, and any unresolved conflict. If evidence is stale or contradictory, block approval or request a fresh lookup. Do not use urgency language that pressures people to approve.
Make decline and defer legitimate options. A reviewer should be able to ask for correction, route to an owner, or stop the run. Record why a high-impact action was declined without shaming the reviewer. Do not let the agent silently retry the same proposal through another tool.
Prevent approval fatigue
Use a risk policy that groups actions into tiers. Low-risk, reversible actions can proceed automatically after testing. Medium-impact actions may need one informed user confirmation. High-impact or irreversible changes may require an authorized reviewer and independent verification. Define tiers with business owners, security, and legal teams rather than inventing them in the prompt.
Bundle related low-risk changes when a reviewer can inspect them together, but never hide individual high-impact actions inside a large batch. Keep the approval summary concise, and expose details on demand. Track approval latency, edit rate, decline rate, and post-approval error rate. A 99% approval rate may indicate excellent proposals, or it may indicate rubber-stamping; sample reviews to understand which.
Preserve an audit trail and test the interface
Store the proposal hash, approver identity, policy version, evidence references, timestamp, and execution result. Minimize retained personal information and set a retention period. Ensure the audit log can answer who approved what and whether the executed action matched the approved one.
Test with deliberate near-misses: a wrong recipient with a similar name, an extra zero in an amount, a permission that is broader than requested, stale evidence, a hidden changed field, and an approval token reused on another action. Ask representative reviewers to complete tasks under realistic time pressure. Measure whether they catch the planted error, not just whether they like the interface.
Human approval works when it is a meaningful decision backed by authority, evidence, and a stable action payload. It fails when people are asked to bless a black box. Design the review around what can go wrong, and let the system proceed automatically only where the consequences and recovery path make that sensible.