Here is a pattern worth knowing about before you buy any AI automation.
Several agent platforms let you attach evaluation criteria to a step, then report an accuracy percentage against those criteria. Convenient. Some of them also offer a button that generates the evaluation criteria from the step's own prompt.
Follow that through. The system writes the instruction. The system writes the test for the instruction. The system grades itself against its own test. Then a number appears on a dashboard, and that number ends up in a slide.
It isn't fraud, and the feature has a legitimate use — it is a fast smoke test during development. The problem is what happens when the output is presented as a measurement of real-world performance. A model that misunderstands your document in a consistent way will write criteria that encode the same misunderstanding, and then score itself 98%.
What a real evaluation looks like
The engineering answer has been standard for decades and does not require anything clever: a held-out labelled set.
- Sample from reality. Pull real items from the client's actual operation — not synthetic examples, not the tidy ones. Include the awkward cases, because those are where money is lost.
- Label them with a human who knows the domain. This is the part that costs real hours, and the part everyone wants to skip. The client's own AP clerk is usually the best labeller in the room.
- Freeze it before you build. The set is written down, its size is agreed, and the pass threshold is agreed — in the contract, before development starts.
- Never train or tune on it. The moment the set influences the build, it stops measuring generalisation and starts measuring memorisation.
- Report the failures, not just the score. "94% on 200 invoices, and here are the 12 it got wrong" is a useful sentence. "98% accurate" is not.
How big does the set need to be?
Bigger than people expect, but smaller than people fear. The intuition that matters is the width of the confidence interval, not the raw count.
| Set size | Observed 95% | Roughly what you can claim |
|---|---|---|
| 20 items | 19/20 | Almost nothing — the interval spans ~75–100% |
| 100 items | 95/100 | "Around 95%, ±4 points" |
| 400 items | 380/400 | "95%, ±2 points" — defensible |
Indicative figures using a standard binomial interval; the point is the shape of the curve, not the third decimal.
For most mid-market document workloads, 150–400 labelled items is the honest zone. Below about 100, you cannot distinguish 90% from 97% — and as the cost arithmetic shows, that difference is worth over a thousand dollars a month in review labour.
Stratify, or you will fool yourself
A random sample from a real inbox is dominated by the easy majority. If 85% of your invoices come from twelve regular suppliers on the same template, a random 200 will be mostly those twelve, and your score will describe a problem you did not need to solve.
Stratify deliberately: sample across suppliers, formats, languages, quality (including the scanned-crooked ones), and edge cases like credit notes and multi-page attachments. Then report accuracy per stratum. It is entirely normal to see 99% on the twelve regulars and 70% on the long tail — and that split tells you exactly where the human queue should sit.
Why this is commercially useful, not just correct
Three reasons we make it a contract term rather than a nice-to-have:
- It survives procurement. "We measured 94% on 200 of your own invoices" is checkable. A vendor's marketing number is not.
- It makes acceptance objective. The engagement passes or fails against a threshold agreed before anyone wrote code. That protects the client from a vendor declaring victory, and protects the vendor from a moving target.
- It gives you a real decision. At your measured accuracy and your volume, does the exception queue cost less than the manual process? Sometimes the answer is no. You should learn that before you spend, not after.
The set stays yours. It gets re-run monthly under a care retainer, because model providers update models, your document mix drifts, and a number from six months ago is a historical artifact rather than a measurement.
The uncomfortable part
Building a held-out set costs hours the client can see and does not obviously want to pay for. It is the easiest line to cut from a proposal, and cutting it makes the proposal cheaper and the project riskier.
We would rather lose on price than deliver an automation whose accuracy nobody actually measured — because the failure mode is not the demo looking bad. It is a system quietly making a few hundred wrong decisions a month while a dashboard reports 98%.