An AI data pipeline is a production line, not a prompt.

01 / DefinitionWhat an AI data pipeline actually is

The second half of that definition is the part most stacks skip. Wiring an LLM call into a script is one afternoon of work, and it genuinely does the semantic step a regex never could. What it does not give you is a pipeline: there is no record of which rows were attempted, no check that what came back is right, and no way for tomorrow's run — or a different machine, or a different agent — to pick up where today's stopped.

The distinction matters because model output is not like transform output. A SQL CAST either succeeds or throws. A model answering "is this company a manufacturer or a trader?" always succeeds — the failure mode is a confident wrong answer that looks exactly like a right one. Machinery that assumes transforms are either correct or loud will wave those rows straight through.

02 / ContrastHow it differs from a classic ETL pipeline

ETL already solved orchestration, retries and scheduling — none of that needs reinventing. What changes is the trust model. A deterministic pipeline that runs green is done; a probabilistic one that runs green has merely finished. The differences concentrate in four places:

Notice what did not change: the destination. Both kinds of pipeline exist to produce tables that downstream tools can join, filter and build on. An AI pipeline that ends in a pile of JSON transcripts has not finished; it has just moved the tabulation problem to whoever reads them next.

03 / AnatomyThe five stages, and who owns each one

Strip any working AI data pipeline to its skeleton and the same five stages appear. Naming them matters less than noticing that ownership alternates between mechanical code and model judgement — and that mixing the two inside one stage is where quality goes to die.

1. Collect. Crawlers, exports, APIs. Land everything append-only: duplicates are a fact about collection, not an error to hide, and a raw table you can re-derive from beats a "clean" one you silently overwrote. No model needed here.
2. Clean. Dedupe by key, normalise encodings, fold obvious variants. Mechanical rules do this faster and more honestly than a model — and every row the cleaner rejects should be written down somewhere, because a row that vanishes without a record is the start of a reconciliation nightmare. Where a model does help with cleaning is its own question.
3. Extract. The judgement step: grow the column the source never had. Manufacturer or trader. Standard clause or risky one. This is LLM data extraction, and it is the stage that makes the pipeline "AI" at all.
4. Gate. Decide, per row and per batch, what gets accepted. Type and enum checks catch malformed output; spot-checks against rows with known answers catch the dangerous kind — well-formed and wrong. Output that skips this stage is opinion, not data.
5. Consume. Accepted values land as real columns in a real table, joined to the source rows they came from — so BI tools, notebooks and the next pipeline can use them without knowing how they were made.

04 / The real questionWhere the state lives decides everything

Here is the test that sorts real pipelines from demos. Kill the process mid-run — close the laptop, drop the session, lose the context window. What survives?

If the answer is "whatever the model remembers", you have an in-context pipeline: the batch cursor, the acceptance decisions and the reasons all live in a window that forgets. It works brilliantly for fifty rows. At five thousand, the early rows have slid out of the window, attention is spread across thousands of rows the model is not currently deciding about, and the failure is quiet — no error, just answers that drift. When the session ends, the run ends with it.

The alternative is server-side state: the pipeline — tables, batches, checks, verdicts, attempts — lives in a service, and model sessions come and go as workers. Any session can ask "what's next?", do a batch, submit it, and disappear. Progress is a fact about the server, not a memory inside a worker. Restart tolerance stops being a feature and becomes a property of the architecture.

This is the design argument behind how Tablize works, but it is not proprietary insight — any pipeline that outlives its sessions converges on the same shape: thin workers, durable line.

05 / OperationRunning the pipeline with a coding agent

The five stages used to require five tools and a human operator. A coding agent (Claude Code, Codex and their peers) collapses that: the agent crawls, cleans, pulls an extraction batch, computes it with the model it already runs on, submits, and asks for the next — while the server keeps the books. In practice the loop looks like this:

# the line and its contract live on the server
tablize import raw_suppliers.csv --new "Supplier listings · raw"
[OK] imported · 115 rows · 5 cols · append-only

# the agent pulls work in batches it can actually hold
tablize batch pull supplier-category --size 25
→ 25 rows · schema: {category: manufacturer | trader | unclear}

# …model does the judgement, agent submits…
tablize batch submit supplier-category ./answers.json
→ 23 accepted · 2 refused (enum violation · golden miss)

Two properties of this loop are worth stealing even if you build your own. First, the agent only ever holds the rows it was handed — context stays small no matter how large the table grows. Second, the model spend rides the flat-rate agent subscription instead of a per-token API bill, which changes the economics of running judgement over every row.

06 / ChecklistA build checklist you can argue with

1. Raw tables are append-only; nothing edits history.
2. Every stage's output is a table with a schema, not a transcript.
3. The extraction schema is declared before the first batch runs — types, enums, required fields.
4. Work is handed out in batches sized to what a model can hold well, not to what it can technically fit.
5. Some rows in every run have answers pinned in advance, and the pipeline refuses batches that miss them.
6. Every attempt — accepted or refused — lands on one ledger with model, batch and verdict attached.
7. Killing every worker session loses no state.

Seven boxes. A stack that ticks all seven is a production line, whoever built it. A stack that ticks five is a good start with two incidents waiting.

07 / FAQQuestions that come up

Adjacent reading, same production-line lens: