AI Can Read a Scanned PDF Until the Table, Footnote, or Minus Sign Matters
Evaluate document AI and OCR on layout, tables, handwriting, footnotes, confidence, provenance, and human review before automating decisions.
Text extraction is only the first problem
Document AI can extract fields from invoices, contracts, forms, and scanned reports. A result may look structured while quietly losing the relationship between a label and its value. Tables can reorder columns, footnotes can disappear, handwriting can be misread, and a minus sign can turn a credit into a charge. A clean JSON response is not evidence that the source was understood correctly.
Teams often start with a few clear PDFs and conclude that extraction is ready. Production documents arrive from different scanners, locales, templates, and generations. Some are rotated, compressed, handwritten, or annotated. The intended use also matters: indexing a document is lower risk than approving payment or determining eligibility.
Define the decision the system supports and the cost of each error. Decide which fields require exact accuracy, when review is mandatory, and whether low-confidence documents should be rejected or routed to a person.
Build a representative corpus
Collect documents with permission and minimize sensitive information. Include layout variations, scans, tables, stamps, footnotes, multi-column pages, handwritten values, poor contrast, multiple languages, and blank or malformed pages. Keep a reference transcription or field labels reviewed by a qualified person.
Measure field-level precision and recall, exact-match rates for amounts and identifiers, table relationship accuracy, page and bounding-box accuracy, and confidence calibration. A document-level pass rate hides whether the model consistently misses tax amounts or contract exceptions. Report results by document source and template family.
Include examples where the answer is absent, ambiguous, or illegible. The system should say “needs review” rather than infer a likely value. Test duplicate pages, superseded versions, and documents that contain conflicting values.
Preserve layout and provenance
For important fields, retain page number and bounding box so a reviewer can inspect the original region. Keep the extracted text connected to the source file, version, and ingestion timestamp. Preserve table structure, reading order, and footnote relationships where they affect meaning.
Do not flatten a complex document into an untraceable text blob before extraction. Use layout-aware preprocessing where appropriate and compare outputs with the original. Validate dates, currencies, identifiers, and totals with deterministic rules. A model can extract a number; code can check whether line items sum to the total.
Store confidence only with a clear interpretation. If a vendor returns a score, test it against actual error rates. A threshold that works for one template may fail on another. Combine confidence with field criticality and validation checks to determine review routing.
Design the human review path
Show the source crop next to each extracted field. Highlight values that fail validation or appear in a low-quality region. Let reviewers correct fields without retyping the full document, and preserve who changed what. For high-impact decisions, use a second reviewer or a separate verification step.
Keep model extraction separate from final business action. A parsed invoice should not automatically trigger payment until vendor identity, amount, duplicate status, and approval rules pass. A contract summary should not replace review of the clauses that govern obligations.
Make errors recoverable. If processing fails, retain the document safely and explain how to retry or request help. Avoid creating duplicate records when a job is retried after a timeout. Use an idempotency key tied to the source document and workflow.
Protect sensitive files
Check provider retention and processing terms before sending documents externally. Restrict access to originals and extracted text, set retention periods, encrypt data in transit and at rest, and prevent sensitive content from appearing in broad logs. Define deletion across indexes, caches, and backups.
Review prompt injection in documents when extracted text will feed an agent with tools. A contract can contain instructions aimed at a model. Keep source text as data, and ensure tool permissions are enforced outside the model.
Document AI can cut repetitive entry, but accuracy must be measured at the field and decision level. Preserve a path back to the page, validate what can be validated deterministically, and route uncertainty to people with authority. The strongest system does not pretend every scan is readable; it knows when the source is not good enough to trust.
Before changing a document template or OCR provider, rerun a fixed sample from every high-volume source. A small form redesign can shift field positions enough to break a parser that relied on layout. Version the extraction configuration and keep a rollback path so operations can return to manual review while the new route is investigated.