01 / RealityHow LLM extraction actually fails
Four modes account for nearly everything that goes wrong at scale, and none of them throw an error:
- Hallucinated values. The evidence doesn't state a founding year; the model
supplies a plausible one. Unless the schema explicitly licenses
null, helpfulness beats honesty every time. - Enum drift. The contract says
manufacturer | trader; row 612 comes back"mostly manufacturer". One drifted value breaks every filter built on the column — quietly. - Confidence illusion. Wrong answers arrive in the same fluent, assured tone as right ones. There is no "uncertain voice" to listen for; any check that relies on the model flagging its own doubt is theatre.
- Batch decay. Judgement quality at row 3 is not quality at row 40. Rows deep in an oversized batch get a skimming reader — same prompt, same model, worse answers.
Because all four modes produce well-formed output, quality in LLM extraction cannot be inspected at the level of single answers. It has to be enforced at the level of process: how much a worker holds at once, what judges every submission, what spot-checks the judgement, and what remembers the attempts. That is the rest of this page.
02 / Handing out workPull-based batches, sized to judgement
The naive loop pushes the whole table at a model session and hopes. The production loop inverts control: the worker pulls a batch, computes it, submits, and asks for the next. Three properties fall out of that inversion:
- Bounded context, at any table size. The worker only ever holds the rows it was handed. Row five thousand gets the same attention as row five — the failure whole-table-in-context pipelines can't avoid.
- Restart tolerance for free. A dead session just means an unclaimed batch. Progress is the server's fact, not a memory inside a worker; any session — or a different agent tomorrow — resumes where the line actually is.
- Parallelism without collision. Two agents pull disjoint batches; the server hands out work exactly once. Scaling out is arithmetic, not a locking design.
Size batches to judgement, not capacity. The context window says 200 rows fit; the pinned-answer hit rate says quality knees at 30. The knee is your batch size — found empirically, and different per column. Capacity-sized batches are how decay becomes policy.
03 / The refereeThe golden gate: enforce, observe, none
Mechanical checks (types, enums, ranges — the declared-checks layer) refuse malformed output. The gate exists for the failure they cannot see: well-formed and wrong. The mechanism is old, borrowed from every industry that ever audited production: salt the work with pieces whose answers you already know.
Pin correct answers for a sample of rows. Hide them inside normal batches — the worker cannot tell which rows are golden. Score every submission against the pinned rows it happened to contain; a batch that misses them is refused wholesale, because a worker wrong on the rows you can check is not more right on the rows you can't.
One gate, three modes — because a line's life has phases:
| Mode | What happens on a golden miss | When it's the right mode |
|---|---|---|
| enforce | Batch refused; worker retries as a new attempt | Production — downstream trusts the column |
| observe | Recorded, not blocking | Calibration — goldens exist but thresholds are being learned |
| none | No spot-check runs | Day one — no pinned answers yet, and the line says so honestly |
The honesty detail that matters: only rows that passed under enforce earn the
accepted mark. A column built under none is still useful — but its
rows say what regime produced them, and nobody gets to launder calibration output as
audited output. Declared checks, meanwhile, judge every submission under every
mode; the gate modes only vary the spot-check's teeth.
04 / MemoryThe attempt ledger — why attempt #2 is a feature
In most extraction scripts, a retry is shame: catch the error, loop, overwrite, pretend it never happened. On a production line the incentive points the other way — a refused batch that silently re-runs destroys the signal that tells you a prompt is marginal, a source shifted, or a model changed underneath you.
So every attempt lands on one ledger: batch, model, verdict, refusal reasons, which attempt finally passed. Attempt #2 is not waste. It is the gate working — and the ledger turns retries into instruments:
# the books, read back
tablize ledger supplier-category
→ attempts: 127 · accepted: 119 · refused: 8
→ refusals: 5 enum · 2 golden miss · 1 null-where-required
→ retry rate trending up since batch 41 — the prompt met a new
listing format; the line noticed before downstream didA rising retry rate on one column is the cheapest early warning in the whole stack. Teams that hide retries buy that warning back later, at incident prices, when a downstream number finally looks wrong and nobody can say which era of extraction produced it.
05 / EconomicsChoosing the model per column, not per project
"Which model should we use?" has a wrong shape — per project. Columns differ:
- Mechanical columns — stated dates, amounts, names copied from evidence — extract cleanly on small, fast models. Paying frontier prices here is pure burn.
- Judgement columns — manufacturer-vs-trader, clause risk, quality signals — genuinely earn a stronger model. This is where accuracy differences are real and downstream-visible.
- The pinned rows referee the decision. Try the cheaper model on a column; if golden hit-rate holds, demote and keep the savings. If it dips, the gate said no — empirically, per column, no vibes involved.
And the spend model matters as much as the model: when a coding agent pulls batches and computes them on its flat-rate subscription, per-column model choice becomes a quality decision instead of a budget negotiation. The server — state, checks, ledger — stays cheap either way. That split, thin workers over a durable line, is the whole Tablize argument in one sentence.
06 / FAQQuestions that come up
Isn't this overkill? My extraction script works.
At fifty rows, it probably does — brilliantly. The machinery here answers questions that only exist at scale and over time: which rows can be trusted, what changed since last month, who checked the work. If nobody will ever ask those, keep the script and this page owes you nothing.
What's a golden gate, in one breath?
Rows with pre-pinned answers hidden in normal batches; miss them and the batch is refused. It catches well-formed-but-wrong output — the failure no schema validator can see — using the oldest trick in quality control: salt the work with known pieces.
Why three gate modes instead of just on/off?
Because pinned answers don't exist on day one. none
says so honestly; observe runs the checks without teeth while you
calibrate; enforce refuses. Rows remember which regime produced them,
so audited and unaudited output never blur.
Why record retries instead of just retrying?
A silent retry looks identical to a first-time pass — which erases the signal that a prompt is marginal or a source shifted. Written down, retry rates per column become the cheapest drift alarm you own. Attempt #2 is the gate working, not the line failing.
Does batch extraction lock me into one model?
The opposite: because batches are judged against pinned answers, you can trial a cheaper model on any column and let the hit-rate decide. Model choice becomes per-column and evidence-based — demote where the gate holds, keep the stronger model where it doesn't.
Adjacent reading, same production-line lens: