[ Demonstration ]

Human-reviewed document extraction: a working demonstration with its failures

A demonstration of the kind of AI workflow we build: software reads invoices, ordinary code checks the result, and a person reviews anything uncertain before it is used. It ran on documents we invented, not on client data.

Prepared by Wagenberg Technology Group LLC, 17 October 2026. Copyright Wagenberg Technology Group LLC, all rights reserved.

Headline (held-out test set, 60 documents)

• Auto-accepted: 34 of 60 (56.7%). Sent to a person: 26 of 60 (43.3%).
• Wrong data that would have passed without review: 0 of 34 auto-accepted documents (0.0%). With only 34 accepted documents, the exact one-sided 95% upper bound on that rate is 8.4%, so this sample cannot show the true rate is lower than that.
• Every field correct: 54 of 60 documents (90.0%). The 6 documents with a wrong or missing field were all routed to review.
• 8 of 26 reviews were unnecessary (every field was already right). That is the price of the conservative routing.

The problem

Accounts-payable teams retype vendor, invoice number, date, PO number, subtotal, tax, total and currency from invoices that arrive in many layouts. Automating that is easy to demo and hard to trust: the costly failure is not a document the system cannot read, it is a document it reads wrongly and accepts without anyone looking. This demonstration measures that number directly and publishes every failure.

Design: where code checks, where a person reviews

1. Extraction (code, deterministic). Rule-based parsing of the document text layer: labelled fields, several date formats, US and European number formats, currency codes and symbols, tolerance for common OCR character confusions in labels. Each field gets a confidence score and records which rule produced it. Vendor names are matched against the buyer's vendor master list; PO numbers are checked against the buyer's own PO formats. No AI model, no network, no paid service.
2. Validation (code). Required fields present; subtotal + tax = total to the cent; effective tax rate between 0% and 30%; date between 2024-01-01 and 30 days after a fixed reference date; duplicate by vendor + invoice number (and a softer check on vendor + total + date); credit notes flagged; positive totals on invoices.
3. Routing (code). A document is auto-accepted only if every one of the 8 fields has confidence of at least 0.85 and every check passes. Anything else goes to the human review queue with the reasons written out. Some amounts are deliberately held below the threshold: OCR-repaired characters (0.60), ambiguous separators (0.50), totals on documents that also show another total-like figure such as a previous balance (0.75), a bare '$' from a vendor that bills in Canadian dollars (0.50).
4. Review (a person). A local web page shows the document text beside the extracted fields, the confidence and rule for each field, the failing checks, and three keyboard-accessible actions: approve, save corrections and approve, reject. Corrected values are re-checked. Every decision, by the extractor or a reviewer, is appended to out/audit_log.jsonl.
5. Nothing is sent anywhere. The outputs are out/accepted.jsonl, out/review_queue.jsonl, out/review_decisions.jsonl and the audit log.

Data

120 invented documents from 14 fictional vendors in the master list plus vendors deliberately left out of it, in 5 layouts (classic invoice, European invoice, supplier-portal table export, statement of charges, plain-text email). Each document has a ground-truth JSON file. Messiness was injected on purpose: several date formats, tax line present, absent or written as 'exempt', missing PO numbers, OCR-style character swaps, duplicate resubmissions in a different layout, credit notes, documents with two totals (page totals and previous balances), printed totals that do not add up, implausible dates, and UK vendors billing in US dollars with day-first dates.

Before any extractor rule was written, the documents were split with a fixed seed into 60 development and 60 held-out test documents (data/split.json). Rules were tuned by looking only at development results. The test set was first scored only after tuning stopped and the code hashes were recorded (evidence/frozen_code_sha256_*.txt). After that, the only change to src/extract.py was opening the audit log in append mode; the extraction output did not change.

Held-out set composition by layout: classic 12, email 12, euro 5, statement 7, table 24. Messiness tags in the held-out set (a document can have several): cad_written_as_bare_dollar 2, credit_note 4, duplicate_resubmission 5, implausible_date 1, no_po 12, no_tax_line 8, ocr_noise 8, printed_total_does_not_add_up 2, tax_exempt_note 7, two_totals_page_total 1, two_totals_prev_balance 2, uk_vendor_billing_usd_dayfirst_date 1.

Results on the held-out test set

The extraction output scored here is byte-identical (sha256 e5097ddcc9cfd41a) to the output scored the first time the test set was evaluated with the frozen code, so these are first-look held-out numbers.

Reading the same documents from their PDF renderings (via pdftotext -layout) instead of the text files gave: all fields correct 54/60; auto-accepted 34/60; wrong data among auto-accepted 0/34. The PDFs are clean digital renderings, not scans, so this only shows the PDF path works.

Every held-out failure, published

A failure is any document with at least one wrong or missing field, or any auto-accepted document that should have gone to a person.

Two of these (DOC-070 and DOC-115) are a plain bug found only on the held-out set. It is left unfixed so the published numbers reproduce from the published code. The fix is to stop the digit-repair step from touching letters inside month names.

Unnecessary reviews (cost, not error)

The 'low confidence date (0.60)' rows on email documents come from the same digit-repair bug: the repair runs over the 30 characters after the date label, so the 'O' in a following 'PO-95594' is "repaired" and the date is marked as OCR-corrected even though it was read correctly. The bug costs reviews and missing dates; in this held-out run it did not produce a wrong accepted value.

Development history (development set only)

• First rules: all fields correct 51/60; auto-accepted 33/60; wrong data among auto-accepted 2/33.
• After fixes made by looking at development failures only: all fields correct 60/60; auto-accepted 37/60; wrong data among auto-accepted 0/37. Fixes: a date-pattern bug, OCR-tolerant labels, PO repair limited to the digits after the prefix, and using the vendor's country (from the vendor master) rather than the currency to read ambiguous dates like 08/03/2026. The last fix removed 2 silent errors on development documents from UK vendors billing in US dollars.
• Final run, development set: all fields correct 60/60; auto-accepted 37/60; wrong data among auto-accepted 0/37.

The development set reached 100% because the rules were tuned on it. The held-out numbers above are the honest ones.

Limits

• Synthetic data is easier than real documents in named ways: the same person wrote the generator and the rules, so the label vocabulary, layouts and even the adversarial cases are ones the rules were designed to expect; there are only 5 layouts; text is clean apart from injected character swaps; no scans, skew, stamps, handwriting, multi-page tables or line items that wrap; every vendor in the master list bills in one currency and one date convention.
• Synthetic data is harder than real documents in other ways: the rate of traps (duplicates, two totals, credit notes, bad arithmetic) is far higher than a normal inbox, and a real accounts team has vendor history, purchase orders and goods receipts to match against, which this demo does not use.
• Small sample. 60 held-out documents and 34 auto-accepted ones. Zero observed errors among 34 is consistent with a true error rate as high as 8.4%.
• Policy choices drive the numbers. Raising or lowering the 0.85 threshold, or the rule that credit notes always go to a person, moves the auto-accept and review rates.
• Not production-ready. No authentication on the review page, no concurrency control, no integration with an accounting system, no retention policy for the audit log, no load testing. This is a working demonstration, not a product.
• No claim is made about time saved for any organisation.

Cost to run

$0. Python 3 standard library for extraction, evaluation and the review page. The optional PDF renderings use the reportlab library if it is installed, and the optional PDF reading path uses pdftotext. No network calls, no cloud or paid AI. One extraction pass over all 120 documents took 0.33 seconds on the machine used (Darwin arm64).

What would change with real documents

• A text layer would come from OCR of scans, so character errors would be more frequent and less regular; the OCR-repair rules here would need to be rebuilt from real error patterns.
• Layout coverage would need to be measured per vendor, and new vendors would start in review until enough of their documents had been seen.
• A labelled sample of real documents, split before tuning exactly as here, would be needed before quoting any accuracy. The synthetic results above do not transfer.
• Matching against purchase orders and goods receipts (three-way match) would add stronger checks than arithmetic alone.
• The review page would need sign-in, roles, and a tamper-evident audit store.
• A learned extractor could be compared against these rules on the same held-out set, using the same headline measure: wrong data among auto-accepted documents.

The code, the documents and the raw evidence are kept privately by Wagenberg Technology Group. For the service itself, see AI workflows and automation. For how we test software and record failures, see the sample launch-readiness review.