The temptation, when an AI-driven classification task looks like one you've done before, is to run it. The prompts are written, the pipeline is built, the universe is clear. Pressing the button on the full run feels like the efficient move.
This is almost always wrong. Not because the prompts are bad — usually they're fine — but because the things that go wrong with large-scale classification tasks are visible in a hundred records, and almost never visible in zero. The cost of running a calibration pass is rounding-error small. The cost of catching a problem only after the full run completes is a redo at full price, plus the time spent on a deliverable nobody can use.
For a financial-data client, the brief was a follow-up to earlier work: classify the same universe of investment firms on two new attributes — a thirteen-category firm-type taxonomy on a couple of thousand records, and a binary regulatory-change flag on the full eleven-thousand-firm universe. The analyst had written both prompts. The instinct was to scale immediately. We didn't.
A calibration pass costs the price of a coffee. A directionally-wrong production run costs the price of running everything twice — plus the cost of the conversation about why the first one didn't work.
What the Sample Found
We ran the analyst's prompts as written on a fixed sample of 100 firms. Two structural problems surfaced immediately, neither of which would have shown up at zero scale.
The firm-type prompt returned "Asset Manager" for over four-fifths of the sample, with high model confidence on every answer. The taxonomy as drafted was simply too coarse — the catch-all category was absorbing genuine variation the analyst would have wanted to see. Fixing it required expanding the taxonomy and adding examples that made the boundary between adjacent categories explicit. Predictable distribution after the fix: roughly half "Asset Manager", twenty percent "Wealth Manager / RIA", and a meaningful tail of more specific firm types that would have been invisible under the original prompt.
The regulatory-change prompt did the opposite. It returned a positive flag on only two percent of the sample, at median confidence well below the threshold the analyst wanted. The prompt was too broad — the model didn't know what was supposed to count, and hedged. The fix was a two-part one: tighten the criteria, then loosen them back when the first refinement over-corrected to zero positives. The landing version returned a thirteen-percent positive rate at high confidence, in the range the analyst's domain knowledge suggested was right.
The Discipline of the Fixed Sample
The technique that makes calibration actually work is unglamorous: every prompt version runs on the same hundred records. Set the random seed once, sample once, and then iterate prompts against that locked sample. When the firm-type distribution shifts from eighty-two percent "Asset Manager" to fifty percent across a longer tail, you know the shift came from the prompt — not from a different draw of accounts. When the regulatory-change positive rate swings from two percent to zero to thirteen, you can compare those numbers directly because the underlying universe is held constant.
Without the fixed sample, prompt iteration becomes a guessing game. With it, every change is a controlled experiment.
The right way to iterate on prompts is the same way you'd iterate on any experiment: change one thing at a time, hold everything else constant, and look at the distribution — not just a few examples.
Justification as a Quality Lever
The other meaningful design choice was asking the model for a brief justification alongside every answer. Returning the answer, a confidence score, and a one-sentence explanation does three things at once.
It gives the analyst a defensible audit trail per record — useful for downstream review, especially on positives that drive action. It improves answer quality by forcing the model to commit to evidence before it commits to a label; hand-waved confident answers become much rarer when the model has to name what it saw. And it makes prompt iteration easier — reading thirty justifications tells you immediately whether the model is reasoning about the right things.
Scaling, After the Sample
Once the prompts were locked, the full production run was uneventful — by design. The pipeline was built to checkpoint after every batch (so any interruption can resume rather than restart), wrap each call in a retry loop with exponential backoff (so transient API errors don't poison a row), and write working files to local disk rather than cloud-synced folders (which slow batch I/O dramatically). None of that infrastructure is novel. It's accumulated discipline from earlier projects.
Across roughly twenty thousand classifications, the production run completed in well under an hour with no recovery passes needed.
Two Lessons
The first is that calibration is the cheapest insurance you can buy on any AI-driven classification project. A hundred records will tell you whether your taxonomy is too coarse, whether your binary prompt is too broad or too narrow, whether the model is hedging, and whether the distribution of answers matches domain expectations. Skipping it to save an hour reliably costs a day.
The second is that the work that prevents problems doesn't show up in the deliverable. The reason the production run was uneventful wasn't that the task was easy. It was that the lessons from previous projects of the same shape — checkpoint after every batch, retry transient failures, never iterate prompts on a moving target, ask for justification by default — had been baked in from the start. The deliverable looked simple. The discipline behind it didn't have to.
Have an AI-driven classification or enrichment project where the stakes are real? Say hello.