AI automation · LLM APIs · Python
Invoice extraction
An LLM reads receipts. Code decides whether to believe it.
Philippine retail receipts extracted to structured records by a language model, then checked by arithmetic the model cannot talk its way past: every line must multiply, lines must sum to the subtotal, and the VAT must be exactly 12/112 of a tax-inclusive total. Failures get one repair attempt with the errors fed back verbatim, then go to a human with the reasons attached.
- Python
- LLM APIs × 4
- SQLite
- Zero dependencies
Repository: invoice-extraction
In short
- A deterministic verification layer between the model and the books — a fabricated total is caught by arithmetic, not by hoping the model was honest
- One repair round: the validation errors go back to the model verbatim, which reliably fixes transcription slips; a second failure routes to a review queue instead of looping
- Anthropic, OpenAI, Groq and Gemini behind one two-method interface with no SDKs — request building is a pure function, so every wire format is unit-tested offline
- 74 tests and an eight-receipt eval harness, all running with no API key and no network
The split that makes it trustworthy
A language model is good at exactly the part software is bad at — reading a crumpled, OCR-mangled receipt — and unreliable at the part software is perfect at: arithmetic. Most AI extraction demos ignore this and pipe whatever the model says into a spreadsheet. This one makes the split absolute. The model produces a structured record; a verification layer then checks every relation the receipt must satisfy regardless of who read it. Each line item must equal quantity times unit price. The lines must sum to the subtotal. On a VAT-exclusive invoice, subtotal plus VAT must equal the total; on the tax-inclusive receipts Philippine retail actually prints, the VAT line must be exactly 12/112 of the total, to the centavo.
Money is integer centavos from the moment it is parsed, and the model is asked for centavos too — ask a model for floats and it will eventually hand back 285.99999999, at which point the verifier is testing the parser's rounding rather than the receipt's arithmetic. Rounding happens in exactly one place in the codebase.
Repair once, then ask a human
An extraction that fails verification is not discarded and not trusted — it goes back to the model exactly once, with the validation errors quoted verbatim: "subtotal 3,630.00 + VAT 435.60 = 4,065.60, total says 5,065.60". Models fix transcription slips reliably when told precisely what disagrees, so schema validation collects every error in one pass rather than failing fast — a model told about one mistake at a time fixes them one retry at a time.
A second failure means the receipt, the model or the rules have a real problem, and looping harder at a real problem burns tokens while hiding it. The record lands in a review queue with the structured data and the reasons attached, because a human reviewing a filled-in form against a receipt is faster than a human starting over.
Four providers, no SDKs, no key needed to test
Anthropic, OpenAI, Groq and Gemini sit behind one interface with two methods. Building a request is a pure function from configuration and prompt to URL, headers and body, so every provider's wire format — down to Gemini taking its key in a header rather than a URL, where it would leak into logs — is unit-tested without a network. The one class that touches the network is eight lines of urllib.
The whole suite and the eval harness run with no API key: a fake provider replays scripted responses, and the harness is calibrated by replaying the golden answers through the full pipeline and requiring a perfect score. Swapping the free-tier Gemini for Claude in production is a config change, not a rewrite.
Seeing it run
Captured from an actual run, not an illustration. The repository has the script that produced it.
$ python demo/run_demo.py 1. The model fabricates a total — off by exactly 1,000 pesos. The kind of error that looks plausible and survives a glance. After one repair round: status: verified attempts: 2 Bayanihan Construction Supply · 2026-0347 · 2026-01-12 · total 4,065.60 The repair prompt named the exact disagreement: - subtotal 3,630.00 + VAT 435.60 = 4,065.60, total says 5,065.60 2. The model cannot fix it — same wrong answer twice. After the repair round fails: status: needs_review attempts: 2 what the verifier rejected: - subtotal 3,630.00 + VAT 435.60 = 4,065.60, total says 5,065.60 -> lands in the review queue with the record and the reasons attached. 3. The queue, as accounting sees it: {'needs_review': 1, 'verified': 1}
Source
- extractor/verify.py
- extractor/pipeline.py
- extractor/providers.py
- evals/run_evals.py