Live Crawling

Live crawling is fetching a web page at the moment a query is made, rather than returning a copy that was crawled and stored earlier. Real-time web access and data freshness describe the property you want; live crawling is the operation a vendor performs to deliver it, and the line item you pay for.

The alternative is an index hit. A search or extraction provider that has already crawled a page can serve it from storage in tens of milliseconds, at effectively no marginal cost to itself, with no chance of the target site refusing the request. A live fetch means an outbound HTTP request at query time, often a headless browser render, frequently anti-bot handling, then extraction – seconds rather than milliseconds, and with failure modes the index never has. The page can be down, geofenced, rate-limited, or newly behind a login. Because neither option is right for every query, vendors ship live crawling as a dial rather than a mode, usually with a fallback rung in the middle: try the cache, fetch live only if the cached copy is missing or older than a threshold you set.

The two most explicit implementations converged on the same shape. Exa's API originally exposed a livecrawl parameter with four settings – never, always, fallback (use the cache, fetch fresh only if nothing is cached) and preferred (favour fresh, accept cache) – and its documentation now marks livecrawl deprecated in favour of maxAgeHours, where a positive value caps how stale a cached copy may be, 0 forces a fresh fetch, and -1 always uses the cache. Firecrawl's maxAge works the same way in milliseconds, defaulting to 172,800,000 (two days), with 0 bypassing the cache. Firecrawl's own documentation promotes maxAge as a 500% speed-up and, in the same place, notes that cached results still cost one credit per page. That second detail is the one buyers miss: on most platforms caching buys latency, not money.

For an AI product, the decision is per query class rather than per product. A pricing checker, a stock or availability lookup, and a breaking-news summariser all need a forced live fetch and should budget for the latency and the occasional failure. Background enrichment, entity lookups, documentation retrieval and anything a user is not watching load can run off a cached copy with a generous age cap. Most production pipelines end up with two or three tiers and a per-tier max age, because the alternative – forcing every call live – makes the p95 latency of the whole product hostage to the slowest site in the result set.

There is a second-order cost. Live crawling moves load onto the publisher at query time rather than at index time, which is exactly the traffic pattern that pushed Cloudflare and others to ship pay-per-crawl and content-signal controls through 2025 and 2026. A live-fetch-heavy architecture is more exposed to that shift than one that reads from a vendor's index, because the block, when it arrives, arrives during a user's request.