Natural-language web scraping in 2026: what changes when the extraction API becomes a prompt

The extraction request used to be a precise thing. You found the page, opened dev tools, and wrote a CSS selector: div.product-price > span.amount. You listed the exact URLs to visit. The scraper did what you told it and broke the moment the site changed a class name. In 2026 that request has quietly changed shape. A growing set of extraction endpoints now take a sentence instead of a selector: "get the product name, price, and in-stock status," pointed at a URL or, increasingly, at no URL at all. The tool works out the rest. This is natural-language web scraping, and it is worth understanding before you wire it into anything that ships.

Natural-language web scraping is extraction where you describe the fields you want in plain English and an AI extraction API returns structured JSON, instead of you writing CSS selectors or XPath against known page structures. It moves the brittle part, the mapping from page to data, from your code into a model. What changes is not the output format but who owns the failure.

What actually shipped in 2026

The shift stopped being a demo this year and became the default interface at several vendors.

Firecrawl replaced its /extract endpoint with /agent. You give it a prompt describing the data you need, and URLs are optional: it can search, navigate, and extract on its own, with the JSON schema itself an optional input rather than a requirement. Under the hood it runs Firecrawl's own reasoning models, Spark 1 Pro and Spark 1 Mini, and offers five free runs a day on dynamic pricing. Notice how the job is described now. The endpoint is no longer "fetch this page and parse it" but "go get this fact."

Firecrawl also shipped /interact on March 25, 2026, which extends the same plain-English contract to actions. You can drive a live browser session by prompt ("fill the applicant name field and submit") or by Playwright code in the same session, handling logins and multi-step flows. State carries between calls and a session stays open for up to ten minutes. It is the extraction shift again, applied to clicking rather than reading.

Newer entrants are built entirely around this contract. Context.dev launched on Product Hunt on July 2, 2026, finishing first that day, out of Y Combinator's S26 batch, positioning itself as one API to scrape, enrich, and understand the web. Apify and other established platforms now expose natural-language "AI extract" endpoints too: pass a URL or raw HTML plus a description of the fields, get structured JSON back. Incumbents and week-old startups are converging on the same interface, which tells you this is where the category is heading. Prompt-based web scraping is no longer one vendor's bet.

If you want the conceptual grounding, this is the productized edge of agentic extraction: a web data extractor that plans its own path to the data instead of running a fixed recipe.

The interface got simpler. The failure mode didn't.

Here is the part the launch posts skip. A selector and a prompt fail in very different ways, and the prompt's way is harder to live with.

A CSS selector is deterministic. When the page changes and span.amount no longer exists, your code throws. The error is loud, it lands in your logs, and it points at the exact field that broke. You fix the selector, review the one-line diff, and move on. The failure is legible.

A prompt is not a contract. Ask a model for "the price" and it will return a price, every time, with full confidence. On a page with a list price, a sale price, and a "members pay" price, nothing throws when it picks the wrong one. The run succeeds. The JSON is well-formed. Your structured output validates against its schema. And the number is wrong in a way no exception will ever surface. This is silent schema drift: the shape stays valid while the meaning rots.

There is also nothing to review. A selector change shows up in a pull request as a diff you can read and reason about. A prompt's behavior can shift because the vendor updated the underlying model, not because you changed a line. Same prompt, same page, a different week, a different answer. For a one-off enrichment job that is fine. For a pipeline feeding a RAG index or an agent that acts on the result, non-determinism you cannot see is a liability you cannot bound. You get the convenience up front and the new class of bug later, and the bug does not announce itself.

The pricing model changed with it

The billing changed alongside the interface, and it is easy to misread. Most of these endpoints meter in credits, and the credit is no longer a stand-in for one HTTP request.

Firecrawl's /interact bills two credits per session-minute, with the initial scrape billed separately at standard rates. That is time-based: a session that stalls on a slow login costs more than a fast one, and you do not know which you have until it runs. Context.dev takes the opposite tack in its pricing, advertising "1 credit = 1 scrape, always," with proxies, JavaScript rendering, and anti-bot bypass folded into that single credit, and it calls out competitors that bill those same capabilities as 5x to 25x credit multipliers. Its Developer tier is $25 a month for 10,000 credits and Pro is $149 for 200,000.

Both models are defensible. The problem shows up when an agent, not a cron job, is spending the credits. A deterministic scraper makes a known number of requests. An agent that "searches, navigates, and extracts" makes as many as it decides it needs, and a prompt-based run that retries or explores can quietly cost several times a simple fetch. When the number of calls is chosen by a model at runtime, your monthly bill inherits that model's judgment. Anyone who has priced this out knows the per-unit rate is the easy half; the hard half is cost per answer, and that denominator is now variable.

How to buy this without regret

Natural-language extraction earns its place in a lot of pipelines. The move is to keep using it and put guardrails around the part that moved into the model. Before it touches production:

• Pin and enforce a schema. Always pass an explicit JSON schema and validate against it on your side. It will not catch a wrong-but-valid value, but it stops shape drift cold.
• Snapshot fixtures for regression. Save real pages and their correct extractions, and re-run them on a schedule. This is the only way to catch the silent drift a passing run hides. Our extraction benchmarks guide covers how to structure that harness.
• Cap credits and set spend alerts. Put a hard ceiling on runs per job and per day. With agentic endpoints choosing their own call count, an uncapped loop is an uncapped invoice.
• Keep a deterministic fallback for high-value fields. For the handful of fields where a wrong answer is expensive (price, availability, identifiers), a boring selector or a validation rule against a second source is worth more than model confidence.
• Test on your pages, not the demo. Vendor examples run on friendly pages. Evaluate on the messy, JavaScript-heavy, login-gated sites you actually target, and measure accuracy, not just whether JSON came back.

If a meaningful share of your extraction is stable and high-volume, it is still worth running the build-versus-buy math: a maintained selector on a page that rarely changes can beat a per-credit model call at scale. Open source is a real option here too, and our Firecrawl vs Crawl4AI comparison lays out that trade-off.

Where this is going

The natural-language interface is here to stay. Describing data beats maintaining selectors for a large and growing share of jobs, and the vendors have decided this is the contract they want to sell. Hype and refusal are both the wrong read. Treat the prompt as what it is: a convenient front end over a probabilistic system that will sometimes be confidently wrong, and price, schema, and test around that fact. The teams that get burned in 2026 will be the ones who mistook an easier interface for an easier problem.