Custom Software With AI Document Extraction: Real Costs, Real Failure Modes

What document extraction actually is once real documents hit it

The demo works like this: you upload a clean PDF invoice, a model returns JSON with vendor, invoice number, date, line items and total, and everyone in the room nods. That demo is real, and it will be running by the end of week one. The rest of this page is about weeks two through nine.

Here is the scene where the naive version breaks. A logistics client sends us a bill of lading. It is a fax. Someone printed a PDF, signed it, scanned it back at 200 DPI on a Brother multifunction, and the scan is rotated 3 degrees with a coffee ring over the container number. The extraction returns a container number that is plausible, well formed, matches the ISO 6346 pattern including check digit, and is wrong by one character. Nothing throws. No error appears in Sentry. The record posts to the customer's TMS, a container gets flagged at the wrong terminal, and three weeks later someone in ops asks why demurrage charges do not reconcile. The model did not fail loudly. It produced a valid looking value with high token probability, because that is exactly what these models do when pixels are ambiguous.

The engineering problem is not whether a model can read a document. It can. The problem is that a document extraction system must know when it does not know, and language models are structurally bad at that. Everything expensive in this build exists to compensate for that one property.

The input is never a clean PDF, and PDFs are not one format

"PDF" describes at least four different engineering problems wearing the same file extension.

Digital-native PDFs have a text layer. You can pull text with pdfplumber or PyMuPDF and skip OCR entirely, which is faster and cheaper by an order of magnitude. Always probe for this first. A large share of what arrives in a typical accounts payable or claims inbox is digital-native, and paying vision-model rates to read text that is already sitting in the file is money set on fire.

Scanned image PDFs need OCR or a vision model. Here you hit skew, JPEG compression artifacts, 200 DPI scans where 300 DPI is the practical floor for reliable character recognition, and staple shadows in the gutter.

Hybrid PDFs are the trap. Some pages have a text layer, some do not, and some have a text layer produced by a bad OCR pass upstream that is confidently garbage. Your probe says "text layer present," you skip OCR, and you extract nonsense. Probe per page, not per document, and check extracted text density against page area. If a page yields under roughly 50 characters but has significant ink coverage, treat it as an image.

Everything that is not a PDF: HEIC photos from phones (iOS default, and half of Python imaging stacks will not open them without pillow-heif), multi-page TIFFs from fax gateways, Office files, and emails where the real document is a forwarded attachment three levels deep in a .msg container.

Then there are tables. Tables are where every extraction project loses two weeks it did not budget. A borderless table with merged cells and a line item that wraps to a second row is genuinely hard, because reading order in the PDF content stream has nothing to do with visual layout. Vision models do better here than classical OCR plus heuristics, but they will still silently merge two line items into one, and your totals will not reconcile. Which brings us to the single most valuable thing in the build.

Arithmetic validation catches more than confidence scores do

Model confidence scores are close to useless for this. Whether you use logprobs from an OpenAI model, or the per-field confidence that Azure AI Document Intelligence and Google Document AI return, high confidence on a hallucinated field is common. The model is confident because the token sequence is likely, not because the value is on the page.

What actually works is validating against properties of the document itself:

• Cross-field arithmetic. Do line items sum to subtotal? Does subtotal plus tax equal total? Does quantity times unit price equal the line amount? This catches merged line items, dropped rows and OCR digit errors better than any confidence threshold. On invoice work, this single check catches the large majority of extraction errors that matter financially.
• Format and checksum validation. IBANs have a mod-97 check. VAT numbers have per-country formats. Container numbers have a check digit. US EINs have valid prefixes. A hallucinated value usually fails the checksum. Validate before you store.
• Grounding by bounding box. Require the model to return the location of each extracted value, then verify that a text or OCR token actually exists in that region. If the model claims a total of $4,820.00 at coordinates where OCR sees "$4,820.00", you have real evidence. If it claims a value at coordinates where nothing is, you have a hallucination. This is the strongest single signal available and most teams skip it because it is more work.
• Cross-referencing your own data. If the extracted vendor tax ID matches a vendor in your database whose name is different from the extracted name, something is wrong. Reconciling against known entities beats trusting the document.

Human review is the product, not the fallback

Nobody quotes for the review queue, and the review queue is where the project lives or dies. There is no accuracy number at which you can delete the human. At 95% field accuracy on a 12-field invoice, roughly 46% of documents contain at least one wrong field. Even at 99% per field, about 11% of documents are wrong somewhere. Per-field accuracy and per-document accuracy are different numbers and vendors quote the flattering one.

So you are building a review UI. That means: the document rendered next to the extracted fields, click a field and it highlights the source region on the page, keyboard-first navigation because reviewers process hundreds a day and a mouse-driven UI doubles their handling time, and a routing rule that decides which documents a human sees at all.

Get the routing rule right and the economics work. Auto-approve when every validator passes and the document matches a known template. Route to review when arithmetic fails, a checksum fails, grounding fails, the vendor is new, or the amount exceeds a threshold. In our builds a mature invoice pipeline lands at 60% to 80% straight-through, with the rest touched by a human for 10 to 30 seconds. That is the actual ROI story, and it is a real one. "Zero humans" is not.

The other thing nobody budgets: reviewer corrections are your most valuable dataset. Every correction is a labeled example of exactly what your pipeline gets wrong on your document mix. Store the before value, the after value, the document, the model version and the prompt version. Without that you cannot prove a prompt change improved anything, and you will make changes that quietly make things worse.

The pipeline is async, and it will fail in ways that duplicate work

Extraction takes 5 to 60 seconds per document depending on page count and model. That is not an HTTP request. You need a queue, and the moment you have a queue you inherit distributed systems problems that a "just call the API" quote does not cover.

Idempotency. A worker times out at 60 seconds, the queue redelivers, the job runs again, and now you have two extraction records and two records posted to the customer's ERP (Enterprise Resource Planning). Key jobs on a content hash of the file, not a UUID generated at upload, because the same PDF will arrive twice from a resend and users will double-upload when the spinner looks stuck. Content hashing also gives you free deduplication and a cache, which is worth real money since re-extracting an identical file is pure waste.

Retry policy that distinguishes failure types. A 429 from the model provider should back off and retry. A malformed JSON response should retry with a repair prompt. A corrupt PDF should go to a dead letter queue and never retry, because retrying it 5 times just burns money on a document that will never parse. Teams that use one blanket retry rule get surprise bills.

Long documents. Page limits are real. Azure AI Document Intelligence caps per-request pages on its custom models. Vision models degrade on very long documents even inside the context window, and a 90-page contract chunked naively will lose cross-page tables. You need a chunking strategy with page overlap and a merge step that reconciles fields appearing in multiple chunks.

Schema drift. You will change the extraction schema. Old records were extracted under the old schema. Version the schema, store the version on every record, and never migrate old extractions by re-running them silently, because the new model will produce different values and someone's reconciled month will unreconcile.

Data handling is a compliance surface, not just a storage question

Documents are the densest PII you will ever handle. An invoice carries bank details. A claim form carries health data. An onboarding packet carries a passport scan and an SSN.

Decisions you must make explicitly, before writing code:

• Does the document leave your infrastructure? If you call a third-party model API, it does. Anthropic and OpenAI do not train on API data by default and both offer zero data retention on request for eligible accounts. Get it in writing if you are in a regulated industry, because "they said they do not train on it" will not satisfy an auditor.
• Residency. EU customers will ask where the document is processed. If your answer is "a US region," expect a stalled deal. Both Azure and Google offer regional endpoints for their document services. Model API regional availability is narrower.
• Retention. Under GDPR, holding the source file forever "for debugging" is not defensible. Decide what you keep: the file, the extraction, or a redacted extraction. Most sane designs keep the file for a defined window (30 to 90 days for dispute resolution), then keep the structured output only.
• HIPAA. If documents contain health data, you need a BAA. Anthropic, OpenAI, AWS Bedrock, Azure OpenAI and Google Cloud all support BAAs on appropriate plans. Your logging pipeline is the usual violation, since the raw document text ends up in a log aggregator that is not covered.

Log redaction is genuinely hard here, because "log the model response for debugging" means logging the extracted SSN. Redact at the logging boundary, structurally, by field name, not with regex over free text.

What this costs and what drives the number

These bands come from Digital Heroes delivery experience across 2,000+ projects.

Focused build inside an existing product, $15,000 to $60,000, 4 to 9 weeks. One or two document families, an extraction pipeline with validators, a basic review queue, and a write path into one downstream system. This is the right shape when you already have auth, storage, a job runner and a UI framework.

Part of a first release, $50,000 to $130,000, 10 to 16 weeks. Same extraction work plus the product around it: upload flows, tenancy, permissions, the review UI built properly with keyboard navigation and source highlighting, an audit trail, and an accuracy evaluation harness.

Full platform, $150,000 to $350,000, 6 to 12 months. Customer-configurable document types, multi-tenant template management, a correction-feedback loop that measurably improves accuracy, integrations into several downstream systems, SLAs, and the compliance posture to sell into regulated buyers.

What moves your number inside those bands, specific to this capability:

• Document variety, not volume. Ten thousand invoices from 3 vendors is a small project. Two hundred invoices from 200 vendors is a large one. Variety drives cost. Volume drives infrastructure cost, which is comparatively trivial.
• Tables and line items. Header-only extraction (vendor, date, total) versus full line-item extraction with reconciliation is often a 2x difference in the extraction workstream.
• Handwriting. Any handwriting on the document adds weeks and lowers your accuracy ceiling permanently. Handwritten checkboxes and signatures are fine. Handwritten free text is a different project.
• Number of downstream systems. Writing to one system you control is easy. Writing into a customer's NetSuite or SAP, with their custom fields and their approval workflow, is a real integration each time.
• The accuracy bar you have to prove. "It generally works" costs one number. "We contractually guarantee 99% on totals" costs another, because now you need a labeled test set, an eval harness in CI, and regression gates on every prompt change. Building the labeled test set alone is often 60 to 150 documents of manual annotation.
• Languages and locales. Date format ambiguity alone (03/04/2026 is two different dates) requires locale inference per document, not a global setting.

Running costs are usually small relative to the build: roughly $0.01 to $0.10 per document at current vision model pricing for a typical few-page document, plus Azure AI Document Intelligence at published rates around $10 per 1,000 pages for prebuilt models and $50 per 1,000 for custom. At 50,000 documents a month you are looking at hundreds to low thousands per month. The build dominates, not the inference.

When you should not build this

If your documents are invoices or receipts and your goal is accounts payable automation, do not build this. Buy Ramp, Bill, Vic.ai or Rossum. They have processed millions of invoices from vendors you have never seen, their vendor template libraries are years ahead of anything you will assemble, and they have already built the review UI you are about to spend $40,000 on. You will not beat them, and the thing you want is a solved commodity.

Same answer for standard identity verification (Persona, Onfido), standard tax forms (already handled by every payroll platform), and generic "turn this PDF into text" where LlamaParse or Unstructured at a few dollars per thousand pages is the whole answer.

Build custom when at least one of these is true: the document type is specific to your industry and no vendor has a prebuilt model for it (carrier rate confirmations, lab result formats, utility interconnection agreements, niche insurance ACORD variants with carrier-specific quirks); extraction is not the product but a step inside a workflow only you have, and shipping documents to a third party breaks the flow; the accuracy that matters depends on your proprietary data, because your ability to cross-reference an extracted part number against your own catalog is worth more than any general-purpose model; or the compliance posture rules out sending documents outside your boundary.

A useful middle path most teams miss: use a managed extraction service as the OCR and layout engine, and build only the domain logic, validation and review on top. Azure AI Document Intelligence or Google Document AI as a layout primitive, your code for everything that makes it correct for your business. That usually lands at the low end of the $15,000 to $60,000 band and is the right call more often than either extreme.

How to brief and vet a developer for this

Brief them with artifacts, not adjectives. Send 30 to 50 real documents (redacted), deliberately including your worst ones: the bad fax, the rotated scan, the one with handwriting in the margin, the 40-page one. A vendor who quotes off a description instead of documents is guessing. Also state your accuracy bar in per-document terms and name what a wrong field actually costs you, because that number decides how much validation is worth building.

Then ask these. The answers separate people who have shipped this from people who have done the demo.

• "How do you detect that an extraction is wrong when the model is confident?" If the answer is a confidence threshold, they have not shipped this. You want to hear about cross-field arithmetic, checksums, and bounding-box grounding.
• "How do you decide whether to OCR a page?" A good answer probes the text layer per page and has an opinion about hybrid PDFs. A weak answer OCRs everything, which means they will overcharge you on inference forever.
• "What happens when the same document is uploaded twice?" You want content hashing and idempotency keys. "We check the filename" is a real answer people give.
• "How will I know a prompt change made accuracy better or worse?" The right answer involves a labeled test set and an eval run before merge. If there is no answer, every future change is a coin flip.
• "Show me the review queue in something you have built." Look for click-to-highlight on the source region and keyboard navigation. If the review screen is a form with no document next to it, reviewers cannot verify anything and the queue is theater.
• "What is your retry policy for a malformed model response versus a corrupt file?" One blanket retry rule signals they have not run this in production long enough to get the bill.
• "Where does the document go, and what do your logs contain?" If they cannot immediately name which third parties see the file and whether raw extraction text hits the log aggregator, you have a compliance problem you will inherit.
• "What accuracy will we get on my documents?" The correct answer is "I do not know until I run your samples," followed by a proposal to run 50 of them. Anyone who quotes an accuracy number before seeing your documents is quoting a benchmark from a vendor's marketing page.

Ask for a paid two-week discovery where the deliverable is a measured accuracy baseline on your real documents, a per-field error breakdown, and a fixed quote for the build. If the accuracy baseline is bad, you learned it for a few thousand dollars instead of finding out in month four.