How Accurate Is AI Document Extraction?

Every vendor of AI document extraction quotes an accuracy figure. None of those figures tells you how the tool will do on your invoices, applications or claim forms, with your scan quality and your edge cases. The only accuracy number that matters is the one you measure yourself.

Short answer: modern extraction is accurate enough to automate most of the typing, but not accurate enough to run unchecked on every field. Measure it on a labeled sample of your own documents, use the confidence score on each field to decide what goes straight through, and send the rest to a person. Set stricter thresholds for fields where an error costs money.

Why there is no single accuracy number

Accuracy changes with the document. A clean digital PDF of a standard invoice is easy. A phone photo of a handwritten form at an angle is hard. The cloud providers say as much in their own documentation. AWS advises users to "provide a high quality image, ideally at least 150 DPI," and not to downsample documents before upload. Microsoft tells users of custom models to "ensure that all variations of a document are included in the training dataset."

Accuracy also changes with what you count. Character accuracy, the share of letters read correctly, can look excellent while whole fields are still wrong, because one wrong digit ruins an amount or a policy number. For business use, measure at the field level: was the due date right, was the total right, was the vendor right.

Accuracy and confidence are different things

Two numbers get mixed up. Accuracy is measured after the fact, by comparing the system's output with the right answers on a set of documents. Confidence is the system's own estimate, returned with each extracted field, of how likely that one field is to be correct.

Microsoft's Document Intelligence documentation defines field confidence as "an estimated probability between 0 and 1 that the prediction is correct." Amazon Textract uses a 0 to 100 scale and describes a confidence score as a number "that indicates the probability that a given prediction is correct."

Confidence is what makes automation safe. It lets you treat each field differently: accept the high-confidence ones and route the uncertain ones to a reviewer, instead of trusting or checking the whole document.

How to measure accuracy on your own documents

Google's Document AI documentation describes the standard method. You label a test set of documents with the correct values, run the processor on them, and compare. It reports three measures:

To do this well, build the test set from the documents you actually receive, including the messy ones, and keep them separate from anything used to set up the system. Score each field type on its own, since a model can be strong on dates and weak on line items. Then decide which errors matter most. A missed field that a person fills in is cheaper than a wrong amount that flows into accounting unnoticed.

Setting confidence thresholds

A confidence threshold is the line between "accept automatically" and "send to a person." Raising it catches more errors but sends more work to reviewers. Google's documentation describes the same tradeoff for evaluation: a higher threshold gives "higher precision, because the predictions are more likely to be correct," and "lower recall, because there are fewer predictions."

The providers give starting points, not answers:

• AWS: "Business processes involving financial decisions might require thresholds of 90% or higher," while archival uses such as handwritten notes "might be as low as 50%."
• Microsoft: "It's best to target a score of 80% or higher," and "for more sensitive cases, like financial or medical records, we recommend a score of close to 100%."
• The AWS tutorial for human review sets an example rule that sends a form to review if the Mail Address key scores below 99 or any key value pair scores below 90.

Set thresholds per field, not per document. A typo in a mailing address is an annoyance. A typo in a payment amount or a coverage limit is a loss. Start strict, measure how often reviewers actually change values, and lower thresholds only where the data shows the system is reliable.

Where human review belongs

Human review works best as a targeted step, not a second full pass. Reviewers should see only the fields that fell below threshold, failed a rule, or belong to a high-stakes category, with the source document beside them. Microsoft's guidance puts it simply: "You can also add a human review stage to validate for more critical automation workflows."

Good places for a person in the loop:

• Fields below their confidence threshold
• Values that fail a validation rule, such as line items that do not add up to the total or a date in the future
• Any field that drives a payment, a coverage decision or a legal obligation
• A small random sample of high-confidence results, to catch errors the model is confident about

That last item matters because confidence scores can be wrong too. Microsoft's guidance notes that when confidence is low on a model that tested well, the documents likely look different from its training data, and the model may need retraining. A periodic sample is how you notice drift before it shows up in your books.

One practical note: AWS has stopped taking new customers for Amazon Augmented AI, its built-in human review service, so new projects on AWS need to build the review step into their own workflow.

Validation rules catch what confidence misses

Confidence is the model's opinion. Rules are facts. Pair them. Totals should equal the sum of line items. Invoice dates should not be in the future. A policy number should match a known format, and a vendor should exist in your system. A field can score high confidence and still fail a rule, and that combination is exactly what a reviewer should see.

Illustrative example: an accounts payable team sends invoices through extraction. Vendor name and invoice number pass through when confidence is high and the vendor matches a record. Amounts must pass a line-item sum check and a high threshold, or a person confirms them. Once a month, a reviewer checks 20 invoices that passed automatically. The numbers in this example are for illustration only and should come from your own testing.

How Lean Discovery Group helps

Our AI document processing work follows the same approach. We build and test against a sample of the documents you actually receive, validate every field against rules, score confidence on each extraction and route anything below your threshold to a person, keep an audit trail of every document, extraction and correction, and feed reviewer fixes back into prompts and rules.

Extraction is often one step in a larger AI workflow automation that delivers the data into tools such as QuickBooks, NetSuite, Salesforce or HubSpot. See the full range of our AI automation services, or book a discovery call and send a sample document.

Frequently asked questions

Sources

• Microsoft Learn, "Interpret and improve accuracy and confidence scores," Azure Document Intelligence, updated April 8, 2026
• AWS, "Best Practices," Amazon Textract Developer Guide
• AWS, "Tutorial: Get Started in the Amazon A2I Console," Amazon SageMaker AI Developer Guide
• Google Cloud, "Evaluate performance," Document AI documentation