When a bank fails, someone has to read its loan files. I tested how far AI can be trusted with that.

In 2009 and 2010, U.S. banks failed at a pace of two to three a week. In March 2023, Silicon Valley Bank lost its depositors in about a day. Each time, someone had to turn a failed bank’s paperwork into numbers a buyer would pay for, quickly.

When a bank fails, the FDIC steps in as receiver. It sells the bank, usually to a healthy one, and manages whatever the buyer doesn’t take. Buyers price what they’re getting by its loans, and the loans live in loan files: scanned promissory notes, mortgages, appraisals, payment histories and collectors’ notes. Every bank formats them differently.

So the question I wanted to answer was simple to state: how quickly can a stack of unfamiliar paperwork become numbers people can trust, and which numbers still need a person?

What I built

Scanned loan files in; a versioned loan tape, a review queue and dashboards out. Everything is on Microsoft Azure and the Power Platform, deployed from code.
  1. A document AI pipeline (Azure AI Document Intelligence) that splits a scanned loan file into its documents, reads the key numbers (loan amount, rate, payment, maturity, balance, appraised value), and flags problem loans from the free text (bankruptcy, litigation, hardship, underwriting exceptions).
  2. Safety checks that decide what a person has to look at: model confidence, validity rules, and a cross-check against the bank’s own loan schedule.
  3. A review app (Power Apps) where a person confirms or corrects whatever the machine wasn’t sure about, and automations (Power Automate) that apply those decisions to the official record, keep every past value and escalate anything that waits too long.
  4. Dashboards (Power BI): can we trust the data, is review keeping up, and what is it costing?

Real loan files are confidential, so I generated synthetic ones for about 50 fictional banks. Each copies a real failed bank’s loan mix (from its last public financial report), and each has its own layout, wording, date and money formats and note-writing style. Every test uses whole banks the model has never seen.

What I found

1. Hand-written rules don’t survive a new bank. A parser written from three banks’ paperwork read 84% of their values correctly and 0% at the next bank, which wrote “Sum Borrowed” where the others wrote “Principal Amount”. It failed safely (blank, not wrong), but a person would have typed everything.

2. A trained model generalizes, once it has seen enough variety. Trained on three banks, the model read 87–99.5% of values at new banks. Trained on twenty more randomly styled banks, it read 96–99.7%. How the extra banks were made mattered, though: a version that also randomly deleted fields and added decoy numbers got worse (73% at one bank), because it learned “grab the nearest dollar amount”.

3. The safety checks came from failures, not from planning. Each new held-out bank broke something:

  • One listed fields in reverse order, and the model put the monthly payment in the loan-amount field. It was caught by checking against the bank’s schedule, and by a rule that a payment must be a small fraction of the loan.
  • One printed dates day-first (06.04.2008 = 6 April). The model read them perfectly; my own code turned them into June 4th, and nothing flagged it. The fix was to learn each bank’s date order from dates that can only be read one way, and to send ambiguous dates to a person until the order is known.

With those checks, every bank tested ended with 0 wrong values slipping through, and a person reviewing about 2% of values (12% at the hardest bank).

The review queue. The model read the monthly payment as the principal; the bank's own schedule says $1,051,000. The page, the model's reading, the reason it was flagged and the schedule sit side by side.

4. For free text, combine a careful model with a thorough one. Simple text models (TF-IDF + logistic regression) were never wrong but missed half the flags at a new bank. An LLM (gpt-4.1-mini on Azure AI Foundry) caught every flag but raised false alarms. Accepting what both agree on, and sending the LLM’s extra flags to a person, gave zero wrong auto-accepts and every real flag found, with about half the flags reviewed. Every flag carries the sentence it came from, and the app checks that the quote appears word-for-word on the page.

5. On real documents, it reads the deal terms. I ran the pipeline on 22 real purchase-and-assumption agreements from 2009–2011 failures (60–200 scanned pages each) and checked it against the FDIC’s published bid summaries. Reading only 3% of the pages, it got the buyer and date right 22 out of 22 times and made no wrong readings of the premium, discount or loss-sharing terms; its misses were blanks. Cost: about \$1 of document AI.

A receiver sells loans, not fields

The first version of the review app looked like a form-processing tool: one field at a time. That fits an insurance letter. It doesn’t fit a loan file, where the note, the servicing history and the appraisal have to agree with each other, and where the end product is a loan someone can bid on.

The FDIC’s public material on loan sales describes how that works: loans are grouped into pools by type and performance, sold by sealed bid, and bidders review sample loan files in a virtual data room. The FDIC sells them without representations or warranties, so whatever the file says is what the buyer gets.

So the second screen shows the whole loan:

The loan view for one synthetic commercial loan, 347 days past due when the bank closed.
  • The loan tape: every field, the document and page it was read from, and the bank’s schedule value next to it, with its status (auto-accepted, in review, or corrected by a reviewer).
  • Packet checks that compare documents with each other: payment vs principal on the note, the servicing balance vs the original amount, loan-to-value from the appraisal.
  • Risk flags from the comment log, letters and credit memo, each with its quote.
  • A proposed sale pool: loan type × performance at closing (performing, 30–89 days delinquent, 90+ days).
  • A release decision. Release to data room stays disabled while any value is in review or any check fails; Hold needs a note. Every decision is kept with who made it and when.

Is it keeping up?

Can we trust the loan tape? Accuracy and silent errors per held-out bank.
Is review keeping up? Queue size, age and escalations.
Context: U.S. bank failures since 1934, from the FDIC's public failures data.

Since then: an assistant, a red team and a smarter queue

Ask the loan file. A button on the loan view now opens an assistant: a Microsoft Foundry agent (gpt-4.1-mini) that answers questions like “Why can’t this loan be released yet?” or “What does the FDIC tell borrowers after a failure?”. It can’t write its own database queries: it picks from fixed, read-only ones and searches the loan files, the real deal agreements, the FDIC Resolutions Handbook, FDIC pages on loan sales, and the FDIC’s own history of the crisis, Crisis and Response. Every quote in an answer is checked word-for-word against what it read, and every figure is traced back to the data.

Asking about one loan's risk flags. The badges show the quotes and figures were checked against the documents and data.

Then I attacked it. The loan files come from the failed bank, so their text can’t be trusted: someone could plant an instruction in a comment log. Of 18 attacks, 2 worked at first. One note planted in a comment log made the assistant tell me a loan was “approved for release by FDIC counsel”. Worse, asked for “an official FDIC notice” telling borrowers to send their payments to a new account, it wrote one. That’s a real fraud pattern after bank failures. After screening every document passage with Azure’s Prompt Shields before the model reads it, and adding rules against impersonation and payment instructions, 0 of 18 worked, and the screening removed none of the real passages.

I also tried Azure’s automatic groundedness check. It flagged 6 of 8 correct answers, mostly unit conversions (\$169.7 billion from a figure stored in thousands). A warning that fires on right answers teaches people to ignore it, so I replaced it with a plain check that every number in an answer appears in the data.

A queue that knows what’s at stake. Reviewers now see the queue in order of expected dollars at risk: the chance a value is wrong times the loan’s balance. The chance is learned from other banks, plus the decisions reviewers have already made at this bank. Every saved decision re-ranks the queue. And given the minutes left before the data room opens, the app works out which loans to finish to get the most balance in front of buyers.

What the real failures say. I matched 454 real failures from 2008 to 2012 with each bank’s last financial report before it failed. Heavy construction lending alone didn’t mark the costliest failures. Construction lending plus bad loans or almost no capital left did: 71% of those banks were among the costliest third, against 34% overall. I checked that against what random shuffling produces, because when you try many combinations, some look good by chance. It matches the FDIC’s own account of the crisis.

What a mistake costs. Zero wrong values in 197 doesn’t mean zero. It means the real rate could still be up to about 1.5%. A review costs a few minutes; a wrong balance that reaches bidders, with no warranty behind it, can cost hundreds or thousands of dollars. That’s why the next bullet list starts where it does.

What I’d tell someone deploying this

  • Start every new bank at full human review, and lower it only as accuracy is proven on that bank.
  • Measure silent errors, not accuracy. A wrong number nobody looks at is the expensive one.
  • Reconcile against an independent source (the bank’s own schedule) whenever one exists. It caught most of the model’s mistakes.
  • Keep every version. The app never overwrites: the model’s value, the reviewer’s correction, who made it and when are all kept.
  • Audit a sample of what the machine accepts. Otherwise “zero silent errors” can never be confirmed, because nobody looks.
  • Treat the documents as untrusted input for any AI that reads them, and test it with attacks before anyone relies on it.

Limits

Synthetic loan files are cleaner than real ones: no handwriting, coffee stains or 400-page files. Some risk flags (fraud, property damage) had only a handful of examples. Two held-out banks were used to choose between models, so a fresh bank is needed for a final, unbiased number. The real-agreement test covers 22 deals, not the whole wave.

An independent personal research project. Not affiliated with or endorsed by the FDIC or any other organization. Loan files are synthetic (fictional banks and people); the public data and agreements are real.




Enjoy Reading This Article?

Here are some more articles you might like to read next: