Nine insurers mailed me a price for my house. Reading them took AI, and knowing when not to trust it.

My renewal said about \$7,200 a year. Nine other insurers had mailed me “personalized quotes” between roughly \$1,000 and \$3,000. Picking the cheap one looked easy. It wasn’t.

No two letters were alike. One says “Dwelling Protection”, another “Dwelling Coverage”. One prints a wind/hail deductible, eight don’t. Two letters for the same house disagreed by almost 20% on how much dwelling coverage it needs. Two had my square footage wrong. To compare them you have to read every letter, pull out the same dozen numbers, and notice which ones can’t be compared at all.

The question: should I switch?

This is the page I wanted at the end: every quote lined up against the current policy, and a plain answer to whether each one is a real comparison.

A quote only counts as comparable when it has the same dwelling coverage, the same deductible and the same wind/hail percentage as the current policy. Anything else gets a reason in the last column (“dwelling differs”, “wind/hail 5%”). Price per \$1,000 of dwelling coverage puts different-sized quotes on one scale.

For my real letters the answer was blunt: none of the nine mailers was comparable, because none of them stated a deductible. So the action wasn’t “pick the cheapest letter”. It was: request binding quotes on my exact coverage, deductible and wind/hail terms, and decide before the renewal date. The gap was large enough to be worth the calls.

That’s a small version of a common problem. Insurers, banks and government offices receive piles of documents in inconsistent formats, and they need structured data they can trust, with a person checking whatever the machine is unsure about. So I built that, on Microsoft Azure, and tested it the way I’d want a vendor to test it for me. Code, results and runbooks: github.com/godot107/insurance-docai.

What I built

1. Read the documents with Azure AI Document Intelligence (scans and digital PDFs).
2. Map every insurer’s wording to one standard list of fields: the six coverages (dwelling, contents, liability…), premium, deductibles, property facts.
3. Check each value: the model’s confidence, a plausible range, and ratios (contents coverage is usually 25–75% of dwelling coverage, for example). Anything suspicious goes to a person.
4. Have a person review the flagged values in a small Power Apps app.
5. Remove personal information before anything is shared.
6. Serve the results through Azure SQL to a Power BI dashboard, and through a Databricks lakehouse for analysis.

Everything is infrastructure as code, uses no passwords or API keys, and costs about nothing while idle.

Testing it honestly

Ten real letters can’t tell you how accurate a system is. So I generated 96 synthetic letters from 8 fictional insurers in 6 different layouts, about half degraded to look like cheap scans. Because I made them, I know every correct value exactly. Two layouts (a postcard and a renewal notice) were held back completely: no model or rule saw them before the test. That’s the question that matters for any document system: how does it do on the format nobody planned for?

The number I care about most is the silent error: a value that’s wrong and nobody was asked to check it. Those are the ones that end up in a report.

The confidence score can lie

Hand-written rules on top of Document Intelligence’s general-purpose model read 96% of fields correctly on familiar layouts, but only 78% on the unfamiliar ones. Worse, the wrong values came with the same confidence (about 0.99) as the right ones. The score means “I read these characters correctly”, not “this number belongs in this field”. A policy of “send low-confidence values to a person” would have let every error through.

A custom model generalizes, and knows when it’s unsure

I trained a custom extraction model on 72 labeled synthetic letters. On the two layouts it had never seen:

Its new weakness: it sometimes fills in a field the letter doesn’t print. But it does that at low confidence, and its confidence actually means something: values above 0.95 were all correct, values below 0.8 only about 30%.

Training on “bad” documents fixed most of that

Why did the first model make values up? In its 72 training letters, every field was always printed, so it learned “there is always a dwelling amount somewhere: find the likeliest number”. Real mail breaks that constantly. So I generated 150 deliberately difficult training letters, each built to teach one thing:

1. Labels above the values, a field left off, and a decoy amount (“Neighbors who switched saved an average of \$1,081”; elsewhere “Last term you paid…”). Lesson: the nearest dollar figure isn’t necessarily the answer, and sometimes the honest answer is “not on this letter”.
2. Fax-style black and white, with the dwelling coverage missing entirely. Lesson: the biggest field can be absent, and thin, broken text still has to be read.
3. A phone photo, upside down. Lesson: orientation and lighting shouldn’t matter.
4. A phone photo at an angle. Lesson: the table doesn’t have to be straight to be a table.

The real-letter walkthrough below shows the same change on actual mail: the first model invented a number where the letter said “Included”; the retrained one didn’t.

The review threshold didn’t transfer, and that’s the real lesson

Every review threshold trades work against risk: review more, and fewer errors slip through. On the test letters the new model looked like it needed only 6.5% of values reviewed. But I’d chosen that cutoff on the same letters I measured it on. So I checked it properly: choose the cutoff on one unseen layout, apply it to the other. On the easy postcard the model barely erred, so the cutoff said “review almost nothing”, and on the harder renewal notice that let 9% of values through wrong.

The fix is a process, not a better statistic: when a new layout shows up, a person reviews its first documents in full, the cutoff is set from those reviews, and only then does the system start trusting confidence scores.

It transfers to real mail

On my own letters, the model trained only on synthetic data read about 97% of printed fields correctly, with no silent errors, at the cost of flagging more fields for review. (These numbers are provisional while I finish double-checking the hand labels.)

Here’s one of them, end to end: a quote mailer that arrived as a phone-quality scan, with my details blacked out.

Six of seven values were read exactly and went straight through. The seventh is the interesting one. The letter prints “Included” instead of an amount, and the two versions of the model handled it differently:

• The first model filled in \$300,000, the liability number from the line above, at a fairly confident 0.83. Confidence alone would have let that through. What caught it was a simple rule: living-expense coverage is normally 5–50% of the dwelling amount, and \$300,000 was well over half of it.
• The retrained model read the word “Included”, reported low confidence, and sent it to a person, who marks it “not on the document”.

Either way, the wrong number never reached the comparison. And note that the mailer doesn’t state a deductible at all, which is why no mailer could be compared like for like with my actual policy.

A smarter router isn’t automatically better

On Databricks I trained a model to predict which extracted values are wrong. It ranked errors far better than the raw confidence score (AUC 0.91 vs 0.53, where 0.5 is a coin flip), yet at the same review workload it didn’t beat a calibrated threshold plus simple rules. With eight insurers, the simple policy wins. Negative results are results.

The person in the loop

The review step is a deliberately simple app, built in Microsoft Power Apps (part of Microsoft’s Power Platform) and hosted there, reading and writing the review queue in Azure SQL. No web server to run: the reviewer opens it in a browser, sees one flagged value at a time, and clicks Accept, Save Correction or Not on document. The app itself is kept as code and deployed from the command line, so it’s versioned like everything else.

Every decision is stored with who made it and when, which turns the review queue into a free model monitor. Replaying the project’s history through it: under the rules, almost every flagged value needed correcting. When the first custom model went live, the share marked “not on the document” jumped from 0% to 59% (it was making values up). After the “bad documents” retraining it fell to 29%. In production, that spike is the alarm that says retrain.

An alarm alone isn’t enough, though. If the reviewers start correcting much more than usual, the system should stop trusting the model before anyone reads an email. The design I’d use is a circuit breaker, set per letter layout:

• Baseline first. When a layout is new, a person reviews its first documents in full anyway. Those reviews set its normal rates: how often values are accepted, corrected, or marked “not on the document”.
• Trip on a real shift, not noise. Compare each window of new decisions with that baseline, and trip only when the shift is well beyond normal variation, with enough decisions to be sure, two windows in a row.
• Fail safe, not shut. Tripping doesn’t stop the work: it stops auto-accepting. Every value from that layout goes to a person while support looks into it. Only a wider failure pauses intake altogether.
• Retrain with a person in the decision. The corrections are new training examples, but a retrained model goes live only after it’s checked on held-out letters and someone approves it. Retraining automatically on a bad week of reviews would teach the model the bad week.

It’s per layout because one insurer redesigning its letter shouldn’t stop everyone else’s.

The dashboard

The same Power BI report has pages for operations and finance next to the homeowner’s page above, all built as code.

Why not just upload the letters to ChatGPT?

Fair question. For nine letters, once, a chat assistant would probably do fine, and I’d use one too. This project isn’t really about my nine letters. They’re a small, checkable stand-in for a problem that looks very different at work:

• Volume and repetition. An insurer, bank or agency gets thousands of documents a day, every day. That needs a pipeline that runs unattended, not a person pasting files into a chat window.
• A measured error rate. A chat answer looks equally sure whether it’s right or wrong. Here every value has a confidence score that was checked against known answers, and the rate of silent errors is a number you can report and manage.
• A person in the loop, on record. Uncertain values go to a reviewer, and every decision is stored with who made it and when. When an auditor asks “where did this number come from?”, there’s an answer.
• The data stays inside your own cloud account. Documents full of names, addresses and account numbers never go to a consumer chat service; access is controlled with the organization’s own sign-in.
• It plugs into everything else. Results land in a database, feed a dashboard, and can feed any other system, instead of living in a chat transcript.
• It extends to other paperwork. Nothing here is specific to insurance. Swap the field list and retrain the custom model, and the same pipeline reads claims forms, invoices, loan files, benefit applications or permits.
• It scales and has a known price. The cloud services scale with volume, and the cost per page is known up front: at list price, reading a page with a custom model costs about 3 cents. Building and testing this whole project cost under a dollar on Azure’s bill so far, because almost everything ran on free tiers (Document Intelligence free tier, the Azure SQL free offer, Databricks Free Edition, the Power Apps developer plan).

The skill on show isn’t “getting an AI to read a letter”. It’s building the system around the AI so that an organization can trust, check and run it.

What I’d tell anyone deploying document AI

• Test on the layouts you didn’t design for. Familiar-layout accuracy (96%) hid the real number (78%).
• Don’t route on raw confidence. Check what it actually predicts first; it may predict nothing.
• Count silent errors, not just accuracy.
• Choose the review workload deliberately, with the trade-off in front of you.
• Check the threshold on data it wasn’t chosen on. Every new layout needs a full-review burn-in.
• Generic privacy tools miss domain IDs. Azure’s PII detector found names and addresses but missed policy and loan numbers; a simple “whatever follows Policy Number” rule caught them.

And for homeowners

Mailers are estimates, not offers. Before comparing prices, match the dwelling coverage, the deductible and the wind/hail percentage, and ask how the roof is paid on a claim. Get binding quotes on the same terms, switch on the renewal date, and repeat every year.

How it fits together

What was actually built: one Azure resource group, deployed from Bicep templates, signed in with Microsoft Entra ID, no passwords or keys.

For a production version (private networking, no public endpoints, separate dev/test/prod), the repo has a reference architecture: docs/ARCHITECTURE.md.

Stack: Azure AI Document Intelligence (layout + custom neural), Azure AI Language, Power Apps, Azure SQL, Power BI, Databricks (Unity Catalog, MLflow), Azure AI Foundry, Bicep, Python, GitHub Actions.

Glossary

• Layout: how a letter is arranged on the page (a table, a postcard, a two-column notice). The same facts can sit in very different places.
• Custom (neural) model: a Document Intelligence model trained on my own labeled examples, instead of the general-purpose one that only reads text and tables.
• Confidence score: the model’s own estimate, from 0 to 1, of how sure it is about a value.
• Calibrated: a confidence score that means what it says: of values scored 0.9, about 90% are right.
• Silent error: a wrong value that no check flagged, so no person ever looked at it.
• Precision: of the values the model returned, the share that were right. Low precision means it makes things up.
• Cutoff (threshold): the confidence below which a value goes to a person.
• Augmentation: making extra, deliberately varied training examples (harder layouts, worse scans) so the model learns the general pattern rather than one look.
• AUC: how well a score ranks wrong values above right ones; 1.0 is perfect, 0.5 is a coin flip.
• Drift: when the documents (or the model’s behavior on them) change over time, so accuracy measured earlier no longer holds. Here, a jump in “not on the document” decisions is the early warning.
• PII: personally identifiable information: names, addresses, phone numbers, policy and loan numbers.
• Bronze / silver / gold: raw data as received → cleaned and checked → ready for reports.
• Infrastructure as code (Bicep): the cloud resources are described in files and created by a script, so they can be rebuilt or torn down exactly.