01 / DefinitionWhat "AI web scraping" actually covers
AI web scraping is any collection workflow where a model does work a human scraper-author used to do. In practice that is one of three modes: the model writes or repairs conventional scraper code; the model reads fetched pages and extracts fields directly, replacing selectors entirely; or a coding agent operates the whole line — fetching, extracting, deduping, and landing rows into a table. The modes differ by an order of magnitude in per-page cost and in what breaks when a site changes.
Vendors blur the three because "AI-powered" sells. Engineers should keep them apart, because each mode has a domain where it is clearly right and two where it is clearly wasteful — and because the mode you pick decides what you maintain: code, prompts, or a contract.
02 / The painWhy classic scrapers break — and what actually changed
A selector-based scraper encodes a bet: this site's structure is stable. The bet loses on a redesign, an A/B test, a locale variant, a lazy-loaded component — and it loses silently: the scraper returns empty strings or misaligned fields, not exceptions. Teams learned to budget more for scraper maintenance than for scraper construction.
What LLMs changed is not that selectors stopped breaking. It is that both sides of the maintenance equation collapsed: a model can re-derive a broken selector from the new page in seconds, and a model can skip selectors entirely by reading the page like a person. The question stopped being "how do we keep the scraper alive" and became "which mode is cheapest for this source at this volume".
03 / The choiceThree modes, honestly compared
| Model writes the scraper | Model reads each page | Agent runs the line | |
|---|---|---|---|
| Per-page cost | ~zero after authoring | Tokens per page, forever | Flat — rides the agent's subscription |
| Site changes | Break it; model repairs on demand | Mostly shrugged off | Repaired in the loop |
| Best for | Few stable sources, high volume | Long-tail or layout-diverse sources | Collection feeding a standing pipeline |
| Weakness | Fleet of scripts to babysit | Cost at volume; subtle misreads | Needs a line to land into |
| You maintain | Code | Prompts + spot checks | A contract: schema, checks, cadence |
Two honest notes on that table. First, mode 1 and mode 2 are converging: the same agent that repairs a selector can fall back to reading pages when repair fails, so real fleets are hybrids. Second, mode 3 is not a different scraping technique at all — it is modes 1 or 2 plus everything the scrape needs to survive: landing, dedupe, extraction, acceptance. Which is the next section's point.
04 / The real storyThe scrape is stage one. Most datasets die in stage two.
Every scraping guide ends where the file lands: listings.json, 4,000
rows, victory. Six weeks later that file is a fossil — nobody remembers which run
produced it, half the rows are duplicates from overlapping crawls, and the one column
anyone actually wanted ("is this seller a manufacturer?") was never in the HTML to begin
with. The scrape succeeded; the dataset still died.
What separates scraped files from scraped assets is what happens next, and it is unglamorous:
- Land append-only. Collection re-visits sources, so duplicates are a fact about collecting, not an error. Overwriting hides them; appending with capture timestamps records them. You can always dedupe an honest history — you can never reconstruct a history from an overwritten table.
- Dedupe as its own step. By a stated key (URL, listing id), mechanically,
with the fold recorded —
115 raw → 62 clean, −53 by urlis an audit line, not trivia. Cleaning has its own guide. - Extract the columns the page never had. The judgement step — LLM extraction over clean rows, batch by batch, with acceptance checks, growing the column the scrape was really for.
- Keep the edge standing. Sources change and rows keep arriving. A line re-crawls on cadence and re-derives everything downstream from raw history; a script makes you re-run the project every quarter.
Ask of any scraping stack: if I re-run it tomorrow, does yesterday's work compound or get replaced? Files get replaced. Lines compound — which is why Tablize treats collection as the first edge of a production line rather than a tool with an output folder.
05 / RulesThe rules of the road don't change because a model is driving
Nothing about AI relaxes the obligations of collection — if anything, agents make it easier to comply, because politeness is one instruction instead of a hand-written backoff loop. The stable ground rules:
- Read and respect robots.txt and published rate limits; throttle to what the site can absorb without noticing you.
- Honour the terms of service of sources you rely on; some data is licensed, not public.
- Collect the minimum personal data you have a basis to hold — regulation (GDPR and its siblings) applies to what you store, not just how you got it.
- Never bypass authentication, paywalls or technical barriers; access control is a legal line, not an engineering puzzle.
- Keep provenance — source URL and capture time per row — so questions about any datum have an answer.
None of this is legal advice; for a specific source in a specific jurisdiction, ask a lawyer. The engineering habit that keeps you safe everywhere is the same one that keeps data honest: write down where every row came from.
06 / FAQQuestions that come up
Do LLMs make traditional scrapers obsolete?
No. For a handful of stable, high-volume sources, selector code — now written and repaired by a model — is still orders of magnitude cheaper per page. LLM-per-page reading wins on the long tail: many sources, diverse layouts, frequent changes. Real stacks mix both and let cost per source decide.
What about JavaScript-heavy sites?
Rendering is orthogonal to intelligence: something still has to execute the page (a headless browser) before either a selector or a model can read it. Agents fold this in naturally — fetch, render if needed, read — but the render cost exists in every mode and belongs in your per-page arithmetic.
Is web scraping legal?
Jurisdiction- and case-dependent; AI changes nothing about it. Respect robots.txt, rate limits and terms; don't hold personal data without a basis; never bypass auth. For a specific source, ask a lawyer — the compliance section above is engineering hygiene, not legal advice.
How do I keep scraped data fresh without re-doing the project?
Make collection a standing edge: re-crawl on cadence, append new captures, and re-derive clean and extracted tables from raw history. Freshness becomes something you read off timestamps. If refresh requires a human remembering a script exists, the data is already stale.
My scrape worked — why is my dataset still unusable?
Because the value was never in the HTML. The column you wanted — category, risk, quality — requires judgement over the scraped evidence. That is extraction, stage three of the line, and it is where scraped files either become assets or stay files.
Adjacent reading, same production-line lens: