AI-Generated Code Can Pass Tests and Still Create a Security Incident
Review generated code for trust boundaries, authorization, injection, secrets, dependencies, and misleading tests with a focused checklist.
Passing tests prove only what the tests cover
AI coding assistants can generate plausible implementations quickly. A patch may compile and pass the existing suite while weakening authorization, trusting user input, logging sensitive data, or adding an unsafe dependency. Generated code often follows common patterns, but common patterns are not automatically correct for your system's threat model.
The pressure to review quickly rises when an agent produces a large diff in minutes. Reviewers may feel they should accept it because the tool claims the tests passed. Treat the patch like code from any contributor: inspect the behavior, trust boundaries, and evidence. The model's summary is a claim to verify, not a security review.
Begin with the requested change and identify assets, actors, and boundaries. Ask what data the code reads or writes, who can invoke it, and which assumptions come from the environment. A small feature can create a broad exposure if it crosses a tenant or privilege boundary.
Review high-risk patterns first
Check authentication and authorization separately. Authentication establishes identity; authorization decides whether that identity may access this resource and action. Look for checks that rely on a model-generated ID or client-side condition. Verify the server checks the current user against the specific object on every request.
Inspect input handling and output encoding. Search, SQL, shell, template, and HTML boundaries need context-appropriate parameterization or escaping. Do not accept a generated regex or command as safe merely because it looks familiar. Validate types, lengths, allowed values, and file paths. Check for path traversal, unsafe deserialization, and server-side request forgery where relevant.
Search the diff for secrets, tokens, private URLs, and debugging output. Generated examples sometimes include hard-coded credentials or log full request bodies. Ensure errors do not reveal stack traces or sensitive record existence. Review telemetry changes with the same care as application code.
Treat dependencies and permissions as part of the patch
Inspect every package addition, version change, install script, and transitive dependency impact. Confirm the package name and maintainer are legitimate; similarly named packages can be malicious. Review lockfile changes and provenance where available. Ask whether a standard library or existing dependency can solve the task with less supply-chain exposure.
Check workflow files, CI configuration, container permissions, and deployment manifests. A code agent may broaden a token scope to make a test pass or disable a security check that blocked it. Verify that the change does not expose secrets to pull requests from untrusted forks or grant write access where read access suffices.
For AI-generated code in an agent workflow, inspect tool permissions and network access. A sandbox should not inherit production credentials. Keep agents from pushing directly to protected branches or changing their own review requirements.
Test adversarial inputs and invariants
Add tests that express the security invariant: one tenant cannot read another's data, a user cannot change an unauthorized record, invalid input is rejected, and an expired token fails. Test negative cases, boundary values, Unicode, oversized content, and concurrent requests where appropriate. Avoid tests that simply copy the implementation's assumptions.
Run static analysis and dependency scanning, then interpret the findings. A clean scanner does not prove the design is safe. If a new endpoint or permission check is introduced, trace it through routes, middleware, data access, and logs. Use a threat model proportional to the feature's impact.
Compare test coverage before and after. A patch that deletes a failing test or loosens an assertion should raise concern. Verify the commands the agent says it ran and note tests it could not execute.
Keep the diff small and accountable
Ask the agent to separate unrelated formatting from functional changes. Review one logical change at a time. Require an engineer to explain why the design is correct and what residual risks remain. If the patch is too broad to understand, split it before merging.
Track post-merge defects by source and task class. Use the pattern to improve prompts, sandbox restrictions, test templates, and human review. Do not use a single incident to declare all AI code unsafe, nor a run of successful patches to remove review.
AI-generated code is an accelerator for implementation, not an exemption from secure engineering. Tests, dependency checks, and a focused review of trust boundaries catch different kinds of failure. A patch is ready when the team understands what it changes and can defend the evidence that it works safely.