TL;DR: In financial services, a wrong extraction creates downstream reconciliation failures, while a null value simply routes to manual review. We built a precision-first invoice extraction engine using a two-tier model architecture: a frontier model generates hyper-specific extraction prompts per layout at onboarding, and a lightweight multimodal model executes them in production. Enforced by three validation gates, the system prioritizes extreme precision over recall.
Supplier Payments Plus (SPP) is BILL's offering for large suppliers who receive payments from many customers on the BILL network. Since SPP is in the business of moving money, we can't compromise on accuracy. When an invoice PDF arrives, we know one thing about it: which supplier sent it. Before anything gets paid, the PDF has to be resolved to two more identities. The first is the specific organization inside the supplier's hierarchy. A supplier can have multiple organizations ("orgs") in our system, and each org has its own bank account. The second is the customer being billed. Resolve either one incorrectly and the consequences are severe: payments reconcile against the wrong account, statements stop making sense, and people stop trusting the system.
That constraint shaped everything about the extraction system inside the SPP Agent. The system is allowed to say "I don't know." It is not allowed to be wrong.
This post describes how we get there. A frontier model writes hyper-specific extraction prompts. A least complex multimodal model executes them at scale. Three validation gates stand between a generated prompt and production to guarantee zero-hallucination extraction before code hits production.
Every incoming invoice is a native-digital PDF, generated directly by the supplier's billing software. Suppliers never send us scans, and we filter any scanned document out of our historical corpora, because extraction from scans is unpredictable in ways that native-digital documents are not.This filter substantially improves extraction reliability, and it becomes essential later, when we build the mechanical hallucination check in Gate 3.
From each PDF, the SPP Agent extracts a fixed set of signals, each with a declared type:
Most of these signals exist to drive matching. When the supplier has more than one org, the extracted supplier signals decide which org the invoice belongs to. When there is exactly one org, that step disappears, since there is only one possibility. The customer signals identify the right customer, and the strongest of them is the account number. It is unique to the customer, it usually follows a fixed format such as a ten-digit code, and a short fixed-format value should be one of the easier extractions on the page. The strongest matching signal is also one of the lest complicated to get right. The invoice number identifies the document itself. The list stops at identity on purpose. Amounts, line items, and due dates are a different extraction problem with different failure modes, and they are out of scope for this post.
Here is the asymmetry the whole design hangs on. A null costs minutes: the invoice drops into a manual matching queue and a person resolves it. A wrong value costs far more, because it does not look wrong. It flows into matching like any other signal and pulls the decision toward the wrong org or the wrong customer. So the target is asymmetric on purpose. Precision is the product; recall is a cost dial. Every prompt tells the model to emit a value only under extremely high confidence, and that null is always an acceptable answer. "Null over wrong" is the closest thing this system has to a motto.
The obvious way to build this is one careful, general prompt: here is an invoice, extract these fields. We went the opposite direction, because of the shape of the input.
Invoices are not arbitrary documents. A supplier emits invoices from a small set of templates, and within one template the documents are structurally identical. Only the values change. So before any model sees a PDF, we run it through a layout detection service. The service is deterministic. It does not use an LLM. It examines the arrangement of content on the page and returns a layout ID, and the same document always produces the same ID. Group a supplier's corpus by layout ID and the problem stops being "read any invoice." It becomes "read this exact form."
That reframing is what makes a low complexity model viable. A prompt written for one specific supplier and layout pair can describe things more predictably. The invoice number sits in the top right corner, inside the box labeled "Invoice #." The customer's address block sits directly below the "Bill To" line. The address in the footer belongs to the payment processor; ignore it. Small multimodal models follow instructions like these reliably. What they do far less reliably is decide which of three addresses belongs to whom on an invoice they have never seen. A generic prompt pushes that judgment to inference time on every document. A layout-specific prompt moves it to onboarding time, where we pay for it once.
The economics follow the same line. The frontier model's work amortizes across every invoice a layout will ever produce. The least complex model is the only per-invoice cost, and it is the least complex multimodal model that can pass our validation gates.
The cardinality works in our favor too. Prompts scale with the number of layouts/formats that each supplier has, not with invoice volume. A supplier runs a handful of templates, so the prompt store grows slowly while the invoice stream it serves keeps growing.
A single universal prompt fails the accuracy bar for the reason above. Hand-built coordinate or regex templates can hit the bar on a good day, and then the supplier nudges a logo and the template breaks silently. Fine-tuning a small model per supplier and layout would mean training infrastructure at an awkward cardinality, and it welds the work to one base model. Prompts are portable. When a less expensive or better small model ships, swapping it in requires no retraining. It requires rerunning the gates. The validation pipeline doubles as our model migration test.
Prompt generation is a workflow of its own. For each supplier and layout group, we hand the most capable model available to us a brief. The brief contains the full list of signals and their data types. It describes where each signal lives, in visual language: "top left corner," "below the total, inside the box with the black border." It specifies the exact output schema, which is not merely a request: at inference time the same schema is enforced as a structured-output constraint, so the model cannot return a malformed response, and every field is either a typed value or an explicit null.The brief states the confidence rule: emit a value only when confidence is extremely high, otherwise null. It also includes the single instruction that yielded the biggest jump in prompt accuracy: explicit framing that the downstream reader is a lightweight model requiring step-by-step instructions.
That last line changes the generated prompts more than anything else in the brief. Without it, the frontier model prompts the way it would write notes to itself: compressed, trusting the reader to disambiguate. With it, the output reads like a runbook for a junior engineer on their first on-call shift. Every ambiguity is resolved in advance. Both addresses on the page are described, with explicit instructions for telling them apart. Every field carries its own null condition. Nothing is left to the reader's judgment, because the reader was chosen for being least complex, not for having judgment.
A fragment of the shape, illustrative rather than from a real supplier:
supplier_phone:
Look in the header band across the top of the page, on the right
side, directly under the supplier logo. The value is a US phone
number. If more than one phone number appears in the header, use
the one labeled "Phone:" and ignore the one labeled "Fax:".
If you are not certain, output null.
Each layout starts with a random sample of roughly 100 documents. People extract every signal from every document in the sample and verify the results by hand. This golden set anchors everything downstream, and its role is easy to misread: the golden set is the durable asset, and any individual prompt is disposable. Prompts get regenerated, sometimes several times. The golden set only grows, as hard documents discovered later are labeled and folded in.
A candidate prompt has to clear three checks, in order. Failing any of them badly means the prompt is discarded and regenerated from scratch. Failing them narrowly means the failures themselves become training material.
We run the least complex model with the candidate prompt against the golden sample and require exact agreement with the human labels. One hundred percent. Not high nineties.
Two reasons. At the volume SPP operates at, a small error rate is not a rounding error; it is a steady stream of misrouted matches. And on a corpus this uniform, where every document is the same form with different values filled in, even one miss is a signal. The prompt contains an ambiguity, and ambiguity compounds across the long tail of the full corpus.
A failure here means regeneration from scratch, not a patch. Patching a prompt against a specific failure accretes special cases until the instructions contradict each other, and small models are exactly the readers who suffer most from contradictory instructions. Regenerating from an enriched brief keeps the prompt coherent.
Passing the golden set proves the prompt works on 100 documents. The full corpus is a harder test. We run a full year of historical PDFs for the layout through the least complex model, then hand every output to the frontier model to audit. The audit is deliberately corpus-level rather than per-document: the frontier model sees all the outputs at once, which surfaces patterns no single document shows, such as a value that resembles a neighboring label; a state code sitting in the city column; nulls clustering across one month of documents; or an invoice number whose format changes partway through the year because the supplier switched numbering systems.
If it finds problems everywhere, which implies the prompt was never right, and so we start over. If it finds a small fraction, those documents become "trouble PDFs." People label them, they join the golden set, and the prompt regenerates against the now harder target. Then the whole loop runs again.
Convergence has a specific definition. The only issues left are legitimate ones: a signal genuinely absent from the PDF, the kind of problem no prompt change can fix. Those documents correctly resolve to nulls and manual matching. When the audit's findings reduce to that category, gate 2 is passed.
Earlier extraction efforts at BILL were frustrated by hallucinations: values that looked plausible, parsed cleanly, and appeared nowhere in the document. A hallucination is the worst case for a precision-first system because it defeats the premise. A null is honest. A hallucination is a confident lie.
Born-digital PDFs give us a mechanical defense. We run every PDF through an OCR service to get the complete document text, then verify that every value the model extracted can be located in that text. It is a string-level check, not a model call. On born-digital documents the OCR output is close to perfect, so the check is trustworthy. A value that cannot be found in the document is a hallucination by definition. No model judgment required.
Grounding has a limit: it proves existence, not assignment. A value can appear somewhere in the document and still be attached to the wrong field, so a grounded output is not automatically a correct one. That is why grounding is the last gate rather than the only gate. Wrong-field mistakes are what the golden set and the full-corpus audit exist to catch; grounding catches the values that were never on the page at all.
The response ladder mirrors gate 2. Many hallucinations: start over. A few: they are trouble PDFs, into the golden set, regenerate, repeat. The target is zero, and grounding turns that from a hope into a property we can check.
Only after a prompt clears all three gates do we consider it validated. It goes into a prompt store keyed by supplier ID and layout ID. Prompts are versioned, and every extraction logs the prompt version that produced it, so any output in production can be traced back to the exact instructions behind it, and a regenerated prompt can be rolled out or rolled back without ambiguity about which version did what.
Next to the generation loop, inference is boring, which is the point. The production pipeline for a single invoice:
The manual queue is not a dead end. When a person resolves an invoice, the resolution is written back to the matching store, so the same signals match automatically the next time they appear. A human handles each novel case once; the system handles the repeats.
Rollout is gated per supplier behind a feature flag, so a supplier can be switched onto the pipeline, and switched back off in minutes if anything looks wrong, without touching the code path anyone else is on.
Today, grounding runs during prompt validation rather than on live traffic. The check is mechanical and inexpensive, so running it on every production extraction is a natural next step. It would move the hallucination check from once per prompt to once per invoice.
Suppliers redesign their invoices. Today, a redesign means the layout service returns an ID we have never seen, every invoice on that layout routes to manual matching while samples accumulate, and the generation pipeline runs once there are enough. It works, but it works by falling back to humans.
When a supplier updates a template, historical transaction data accelerates re-onboarding. A new layout from an existing supplier arrives with established customer account numbers and contact details. By matching these known entities against the new layout, the system automatically bootstraps candidate golden data, significantly reducing manual labeling time.
To evaluate prompt quality and system stability, we track a strict operational scorecard: Gate 1 first-pass accuracy on golden sets, average regeneration loops per layout, production null rates, and zero-hallucination verification at Gate 3. By treating prompt validation as a deterministic software pipeline, we maintain near-100% precision across incoming documents.
Conclusion
When building AI for financial services, accuracy is not the best metric to optimize, it is a must have requirement. Designing backwards from the cost of a false positive forced us to build an architecture where null is always an acceptable answer. By shifting intelligence to onboarding, enforcing strict structural boundaries, and verifying every output against underlying OCR source text, we turn non-deterministic vision models into predictable, production-grade systems.
