Why we don't show confidence scores

Open any document-extraction tool and you'll find a small percentage next to each value it produces. 0.92. Sometimes shaded a reassuring green. It looks like information.

I want to argue that it isn't — at least not information you can use — and that building it anyway taught us what to build instead.

What can you do with 0.92?

Suppose the tool read a bid price off a scanned proposal and tells you it's 92% confident. What is your next move?

You can't ship the number — 8% doubt on a price is not a rounding error, it's a different bidder winning. You can't dismiss it either; 92% is usually right. The score doesn't tell you whether this value is wrong, only that some values like it sometimes are. So you do the only sensible thing: you open the PDF and check. The percentage changed nothing about your afternoon.

Worse, it changes something about your habits. A score invites you to read "0.92" as "probably fine" and skip the check — and the values most likely to be wrong are precisely the ones that looked plausible enough to pass a skim. The failure mode of a confidence score is not that it's inaccurate. It's that it talks you out of verifying at exactly the moment verification was the whole point.

The honest version of confidence

Here's the thing we kept coming back to while building Docabra: when a person checks a value, they are not asking "how sure are you." They are asking "where did you read this." The first question has no actionable answer. The second one does.

So the honest form of confidence is not a number, it's provenance. Either the tool can point at the exact words it read — the page, the sentence — or it should admit it can't. A value it can't point to is a value you check yourself. That's binary, and it's actionable: instead of reviewing ninety-two percent of a table, you review the cells with no source attached, which is usually a handful.

We call this click-to-source: every cell keeps the words it was read from, and clicking the cell brings those words up, highlighted on the original page. Scanned document or born-digital, same thing. There's a working example with six real proposals if you want to feel the difference rather than read about it.

Checking is the job

The deeper lesson was about what extraction is for. Nobody wants a table of values; they want a table they can sign off on. The tool's job is not to be right so often that you stop checking — no tool gets there, and pretending otherwise just moves the errors to where you won't catch them. The job is to make checking cheap enough that you actually do it: blanks first, outliers second, and every value one click from its evidence.

Confidence scores optimize for the demo. Provenance optimizes for the second Tuesday, when someone on the evaluation committee asks where a number came from and you have an answer in two seconds instead of an afternoon.

That's why there's no percentage next to our cells. There's a door back to the page instead.