Document AI and OCR Today — How an LLM Ends Up Reading Your Invoices

How machines came to read invoices and scanned PDFs, from first principles: text detection and CTC, how errors are measured (CER), LayoutLM's trick of embedding coordinates alongside words, the OCR-free Donut line, and today's habit of handing the page straight to a VLM — plus the walls that matter in production: tables, handwriting, and hallucination.

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

"Reading" is really three jobs

Hand someone a box of invoices and ask them to total up the amounts. What they actually do splits into three jobs: find where ink sits on the paper, work out which characters those marks are, and decide whether a given string is the invoice total or the wire transfer fee.

Document AI gives each of these a separate name. The first is text detection, the second is text recognition, and together they are OCR (Optical Character Recognition). The third is document understanding.

OCR is old — research goes back to the 1960s, and reading cleanly printed characters has been a solved problem for a long time. Yet complaints that "we still can't automate invoice entry" never went away, because the thing that jammed was almost always the third job. The characters come out fine. What they refer to does not.

Why forms are harder than prose

In a novel, reading left-to-right, top-to-bottom recovers the meaning. Nearly all of it lives in the order of the words.

A form is not like that. You know the "$1,320.00" in the bottom right is the grand total because it is in the bottom right. The same number partway down the line items would be a per-item subtotal. The understanding that a header sits at the top of a column and governs every cell beneath it rests entirely on two-dimensional placement. On a form, layout itself carries meaning.

That is where the trouble starts, because the moment you flatten the characters into a single stream, the meaning breaks. Anyone who has pulled text out of a two-column PDF and gotten left-column and right-column lines interleaved into nonsense has seen it. The deep difficulty in document understanding is less about recognition accuracy and more about what gets destroyed when two dimensions collapse into one.

Classical OCR: detect, recognize, post-process

Through the 2010s the standard build had three clean stages. Text detection first: find the regions where characters sit and cut them out as rectangles (bounding boxes). It's essentially object detection, with "line of text" as the object class.

Then line recognition. Turn the cropped line image into a horizontal sequence of features with a convolutional net, and emit a string from that sequence. The awkward part is that you don't know in advance which horizontal slice corresponds to which character. How many pixels the H in HELLO occupies depends on the font and the letter spacing.

CTC (Connectionist Temporal Classification) solved this by introducing a special "blank" symbol and summing over every alignment that could have produced the target.

Here $\mathbf{x}$ is the feature sequence from the line image, $\mathbf{y}$ is the target string, and $T$ is the number of horizontal slices. A $\pi$ is one particular assignment of a character (or blank) to each slice, and $\mathcal{B}$ is the operation that collapses repeats and strips blanks. Equation (1) just says: enumerate every assignment that collapses to $\mathbf{y}$, and add up their probabilities.

Put in words: the score for "this line reads HELLO" is the total weight of every way those five letters could have been stretched across the slices — the H covering three slices in one reading, five in another, the double L spread differently again. Nobody has to label where each character starts — that was CTC's contribution.

Finally post-processing: apply dictionaries and regular expressions for "looks like a date" or "looks like an amount", and repair the classic confusions between O and 0, 1 and l.

How you measure whether it read correctly

OCR is scored not by accuracy but by edit distance — how many character substitutions, deletions, and insertions it takes to turn the output into the reference.

$S$ is the number of substituted characters, $D$ deletions, $I$ insertions, and $N$ the length of the reference. Equation (2) says "how many repairs per reference character", and it is called CER (Character Error Rate). Count in words instead and you get WER.

Stated plainly, it is the fraction of the page you would have to retype: a CER of 0.05 means five characters' worth of correction for every hundred characters of ground truth — which says nothing at all about which five, and that turns out to matter enormously.

Watch this number alone and it will eventually burn you. CER can be comfortably low while the one wrong character is a digit in an amount, which stops the business cold. What production actually needs is field-level accuracy, not character-level accuracy. "What fraction of invoices had the grand total extracted correctly" is a far more decision-ready number.