Web Data Vendor Due Diligence: The Questions to Ask Before You Sign

When you buy web data for an AI product, you are paying for an action rather than a file. Someone fetches a page that belongs to someone else, and the question that decides your risk is which legal entity did the fetching, under whose terms, and what they kept afterwards. Everything else in a due-diligence pack (the SOC 2 badge, the uptime SLA, the DPA) sits downstream of that.

Six questions follow from it, ordered by who carries the risk when a publisher complains: which entity performs the fetch, which security attestations are current and what they cover, how long queries and fetched pages are retained, who the subprocessors are and where fetches originate, whether the vendor can produce per-record provenance, and what the indemnity and exit terms actually cover. Most published checklists in this market come from the vendors themselves, which makes them arguments. The version below is the one none of them can publish, because any vendor that publishes it has to apply it to itself.

Two dates make this worth doing now rather than at renewal. From 2 August 2026 the European Commission's AI Office can enforce the AI Act's training-data summary obligation against providers of general-purpose AI models, which turns provenance from a nice-to-have into a document a buyer can reasonably demand from a supplier. And Cloudflare's July 2026 announcement of default crawler policies means the access your vendor relies on may be repriced or blocked by an intermediary you have no contract with. Both point the same way: get the answers in writing while the vendor still wants your signature.

If you have not yet decided whether to buy at all, the technical and cost side of that call is in build vs buy for web data. This guide starts after that decision, at the point where you have a shortlist and a draft order form.

Who is on the hook when your agent fetches someone else's page

There are two contract shapes in this market and they allocate risk in opposite directions.

You supply the URL. This covers extraction APIs and browser infrastructure: Firecrawl, Browserbase, Airtop. You pass a target, the vendor's infrastructure performs the fetch. The vendor is the technical actor, but nearly every set of terms in this shape makes you responsible for having the right to target that URL. Read the acceptable-use section: it usually contains a list of things you promise not to fetch, and that list is the vendor's liability shield.

The vendor supplies the corpus. This covers search APIs and pre-collected datasets: Exa, Brave Search API, Diffbot, Common Crawl. Here the collection decision was the vendor's. You are buying access to something already gathered, so the provenance question moves to them and the question you should be pressing is what rights come attached to the results.

The distinction sounds academic until a publisher's lawyer sends a letter. Ask each shortlisted vendor to state, in one sentence in the order form, which of the two it considers itself. Vendors that sell both (an index plus a fetch-this-URL endpoint) should say so per product, because the answer differs by endpoint.

Two things worth watching in the case law as of July 2026. A federal court in the Northern District of California granted SerpApi's motion to dismiss Google's DMCA claims on 20 July 2026, per SerpApi's own announcement and reporting from PPC Land and Search Engine Journal; the court gave Google 21 days to amend on a narrower set of claims involving results containing copyrighted snippets, so this is not a settled outcome and a separate Reddit suit naming SerpApi remains live. Separately, Press Gazette reported that Digital Content Next sent Common Crawl a cease-and-desist on 3 June 2026 on behalf of members including AP, the New York Times, NBC, Bloomberg, NPR and Fox, demanding it stop retaining and sharing their content and delete what it holds. DCN's lawyers also said they were reviewing whether Common Crawl's earlier statements about honoring removal requests were accurate; that is an allegation, not a finding. Neither matter has resolved. Both are reasons to ask a vendor what happens to your pipeline if its source is ordered to delete an archive.

Certifications that carry weight: SOC 2 Type II, ISO 27001, HIPAA

A badge on a pricing page is not an answer. Ask what the badge covers.

The two follow-up questions that separate real answers from marketing: what is the audit period, and what is the scope. A SOC 2 Type II covering only the control plane while the fetching fleet sits outside scope is common and mostly useless for this category.

Among directory vendors, published postures vary by segment. Browserbase has written publicly about reaching SOC 2 and HIPAA compliance and documents enterprise security controls; ask which SOC 2 type is current, because Type I and Type II are different assurances and vendors do not always distinguish them in marketing copy. Airtop markets SOC 2 and HIPAA as part of its cloud browser product. Exa publishes a security reference stating SOC 2 Type II, with HIPAA offered on enterprise plans. Firecrawl states SOC 2 Type II with a DPA available. PromptCloud, which sells managed extraction with human QA rather than a self-serve API, states ISO 27001. Treat all of these as vendor-published and ask for the underlying report under NDA before you rely on any of them.

Enterprise extraction vendors like Zyte and pipeline-oriented platforms like Nimble sell into procurement processes where this paperwork is routine, which is often the real reason to pay their prices rather than a self-serve API's. If your buyer is a bank or a hospital system, that overhead is the product.

Retention: what happens to your queries and to the pages you fetched

Retention is two questions that vendors frequently answer as one.

Your queries. A search API sees every string you send it. If your agent searches for a customer's company name, an unreleased product codename, or an internal ticket ID, that string is now in someone else's logs. Ask how long raw query text is retained, whether it is retained in identifiable form, and whether it feeds ranking, evaluation datasets, or model training.

The pages fetched for you. Extraction APIs cache aggressively because caching is how they hit their price points. That cache is a shared asset. Ask whether content fetched on your behalf is served to other customers, how long it lives, and whether you can opt out at the cost of a higher per-request price.

A third, quieter question: derived artifacts. If the vendor embeds your fetched content into a vector store to power semantic dedup or reranking, deleting the source rows may not delete the embeddings. Any deletion clause that does not name derived data is incomplete.

Zero-retention modes exist in this market but are usually enterprise-tier and priced accordingly. If you need one, raise it before the demo, because it often changes which plan you are being sold.

Subprocessors, regions, and where the fetch actually originates

Every vendor in this category is a stack of other vendors. The API you sign with is typically running on a hyperscaler, resolving CAPTCHAs through a third party, and routing egress through IP suppliers it does not own. That last one matters: a vendor can hold every certification listed above and still route your fetches through residential IP pools whose consent model you cannot inspect. Proxy-first vendors are not in this directory, but they sit underneath a good deal of it, and the subprocessor list is where they become visible.

Three things to request:

• The current subprocessor list, with a commitment to notify you of changes and a window to object. This is standard in a DPA and unusual to see enforced.
• Egress regions. Where does the request leave from? This determines which jurisdiction's rules apply to the fetch and whether a geo-restricted target is being accessed from a region it blocks.
• IP sourcing. Ask directly whether residential or mobile IPs are used, and if so, how the supplier obtains consent. A vendor that cannot describe its own IP supply chain has outsourced a risk it has not priced.

For data residency, ask where fetched content is stored rather than where the API is hosted. They are frequently different, and the storage location is the one that appears in a regulator's question.

Provenance: can the vendor tell you where a corpus came from

A vendor that collected the data itself can usually answer this; one that bought a corpus and relabelled it often cannot. At minimum you want, per record: the source URL, the fetch timestamp, and the robots.txt status at the time of fetch.

Knowledge-graph vendors are structurally better placed to answer this than bulk dataset sellers, because the link back to the source page is part of how the product works. Diffbot builds entity records by parsing pages, so asking for the source URL behind a fact is a reasonable request rather than an unusual one. A vendor selling a flat corpus with no per-record origin is asking you to trust a summary.

Ask three specific things:

1. Can you produce, for a sample of 100 records, the source URL and fetch date?
2. What is your process when a publisher requests removal, and what is your median time to complete one?
3. Have you ever been unable to complete a removal request, and why?

The third question is the useful one. The Common Crawl matter, as reported by Press Gazette, turns partly on the gap between saying a removal was honored and completing it at scale. Any vendor holding a large archive has this problem; the ones worth buying from can describe it.

If your product trains or fine-tunes on this data rather than retrieving from it at query time, the provenance bar is higher again. That case is covered separately in web data for LLM training.

The EU AI Act training-data summary and how it lands on your suppliers

Article 53(1)(d) of the EU AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of the content used for training. The Commission's AI Office published the mandatory template on 24 July 2025. It asks for data sources by type, including publicly available datasets, licensed content, user data, synthetic data, and web-scraped data, and per the template it requires disclosure of the top 10% of domain names crawled. Summaries must be updated at least every six months or on material change.

The enforcement date is 2 August 2026. From that point the AI Office may verify compliance and issue corrective measures, with penalties under Article 101 reaching €15 million or 3% of global annual turnover, whichever is greater.

Two things follow for a buyer.

First, know whether the duty binds you. It binds GPAI model providers. Most teams building a RAG pipeline or an agent on top of someone else's model are not providers, and should resist vendors who imply otherwise in order to sell a compliance tier. If you fine-tune and distribute a general-purpose model, get legal advice on where you sit.

Second, and regardless: the duty makes the question answerable. Suppliers who sell into GPAI providers now have to be able to describe their crawl by domain, and a supplier that has built that reporting for one customer can run it for you. So the question is cheap to ask. If a vendor cannot say which domains are in its corpus in July 2026, its own largest customers are about to ask for the same thing with a regulator behind them.

Robots, licensing terms and the vendor's declared crawler policy

Ask for the vendor's crawler user-agent string, and ask whether it honors robots.txt by default, on request, or not at all. The answers differ legitimately: a vendor fetching a URL you explicitly supplied is in a different position from one crawling broadly to build an index. What you want is a stated policy, not a shrug.

Then ask about rights on the way out. Brave Search API is the clearest illustration in the directory that usage rights are a priced feature rather than an assumption: its published plans distinguish tiers by what you may do with the results, with AI-app usage rights attached to specific plans (Brave's pricing page lists an AI-usage tier at $9 per 1,000 requests as of July 2026; confirm the current figure before budgeting). Most buyers assume that paying for results buys the right to put them in a product. Read the terms and check.

The intermediary layer is now part of this. Cloudflare's July 2026 press release describes a two-month consultation ending 15 September 2026, after which its stated defaults will permit search but block training and agent use on pages carrying ads, and block mixed crawlers that do not let site owners distinguish between search, agent use and training. That is Cloudflare's own description of its plan rather than a shipped state of the world, and the details may move. But the shape of the risk is clear: a vendor whose crawler is classified as mixed could lose access to a slice of the web without either of you doing anything. Ask what the vendor's classification plan is.

Indemnification, active litigation and continuity clauses

Read the indemnity and the liability cap together, because a strong indemnity under a cap equal to three months of fees is a gesture.

• Scope. Does the indemnity cover IP claims arising from the vendor's index, from URLs you supplied, or both? These are very different products.
• Cap. Is the IP indemnity carved out of the general cap, or subject to it?
• Survival. Does it survive termination? Claims about data collected in 2026 can arrive in 2028.
• Disclosure. Ask directly: is the vendor a party to any active litigation concerning data collection? The answer is public for some vendors in this market and unasked for most.

Continuity is the clause buyers skip, and this category consolidates quickly. Tavily was acquired by Nebius in February 2026 and kept operating, which is the common outcome. Even so, an acquisition is when pricing, retention policy and roadmap get rewritten. Ask for a change-of-control notice provision and a price-protection window.

SLAs, rate limits and a workable exit ramp

An SLA that promises API uptime while your actual failure mode is a target site blocking the vendor is measuring the wrong thing. Push for a success-rate commitment on your specific target set, measured over a defined window, with a credit attached. Vendors will resist, and how they resist tells you how confident they are.

Rate limits: get the concurrency ceiling, the burst allowance, and the process for raising them, in the contract rather than in a docs page they can edit. Ask what happens at the ceiling, because a queue and a 429 have different consequences for an agent a user is waiting on.

The exit ramp is four clauses: a notice period binding both parties, an export in a format readable without the vendor's tooling, a deletion certificate covering queries, cache and derived embeddings, and a price-change term with a cap and a notice window. The last one is not boilerplate here. Per-request rates in this category live on a pricing page the vendor can edit, and free tiers and credit grants have a habit of shrinking once a product leaves its growth phase.

A one-page questionnaire to send before the demo

Send this before the call. The response time and the specificity are themselves data.

Fourteen questions, and a vendor that answers all of them in writing within a week is telling you something a demo cannot. The ones that matter most are 1, 3, 8 and 12: who fetched, what was kept, can you prove where it came from, and who pays when it goes wrong. The rest are how you find out whether the first four answers are true.