The Demo Read the Clean PDF. Production Sent a Fax.

A document-reading demo may work perfectly on a clean PDF. Production may send a fax that's skewed, blurred, or broken by compression.

I want to know which documents earned an extraction score. Suppose an extractor returns “04/12” as a discharge date. The characters appear on the page, so a check searching for that string passes.

It's the admission date.

The OCR read the characters correctly, and the result is still wrong. Someone put them in the discharge field. That's the failure. It's like filing the right page in the wrong folder.

The field label says which date belongs there. Finding the same characters elsewhere on the page isn't enough. Keeping the page location beside the result lets a reviewer see the mistake without hunting through the file.

The useful error report says the extractor returned the admission date for the discharge field on this faxed form. Now the team knows what to repair.

IMO, the test set needs the mess that arrives in the real workflow. Rotated pages. Handwritten additions. Faint copies. Faxes where compression removed part of a character. The people responsible for those records should decide which samples may enter the test environment and who may see them. A purpose-made form with fictional values can exercise a layout defect without moving real records into the wrong place.

On a page where one digit can't be read, the expected result can be “unreadable: needs review.” A clearer original may later establish what the damaged copy lost, but that is different from recovering the value from the damaged copy.

The extraction service can handle that case correctly and the overall system can still fail. A later step may replace “unknown” with a default date. Now a later step has put information into the final record that never appeared in the source.

I'd follow the field all the way to the stored or exported record. Stopping at the extractor's response stops before the invented value appears.

Amazon Textract recommends confidence thresholds and human review for lower-confidence results in sensitive uses. That can help direct attention to uncertain extraction. It can't establish that a confidently read admission date belongs in the discharge field.

For this example, I'd keep unreadable images separate from correctly read values assigned to the wrong field. The repairs are different. Some samples should stay out of tuning so the next version has to answer to documents it wasn't adjusted against.

When a production fax exposes a new failure, I'd add an authorized copy and its expected result to the test set. The next version has to read that fax correctly before the fix counts.

Source: Amazon Textract best practices.