What is the simplest way to convert a URL into clean markdown for an LLM?

For a one-shot conversion, prepend `https://r.jina.ai/` to the URL. The Jina Reader API returns LLM-ready markdown with no auth needed for low volumes. For production pipelines, run Trafilatura or Mozilla Readability locally if you can fetch the HTML, or call Firecrawl, Crawl4AI, or Jina Reader as a service if you need JS rendering or anti-bot handling.

When should I use a hosted API vs. a local library for markdown extraction?

Use a local library (Trafilatura, Readability) when you can fetch the HTML yourself and the site doesn't require JS rendering or anti-bot bypass. Switch to a hosted API (Firecrawl, Jina Reader, Crawl4AI Cloud) when you need rendered content from sites with anti-bot defenses, or when you want pre-processed extraction without managing scraper infrastructure.

What is the difference between extraction and crawling?

Extraction converts a single URL into clean content. Crawling discovers and extracts many URLs from a starting point – following links, respecting robots.txt, deduplicating. Most of the tools below do extraction; Firecrawl, Crawl4AI, and ScrapingBee also support crawl modes for full-site ingestion.

How do I keep extraction quality consistent across thousands of sites?

Don't expect a single library to win everywhere. Run a sample of your target domains through two or three extractors and pick the one with the highest content-recall on your specific corpus. The Web Content Extraction Benchmark methodology is a useful template. If accuracy matters more than cost, agentic extraction tools handle long-tail layouts better than rule-based extractors.

What about JavaScript-heavy sites?

Trafilatura and Readability work on the HTML you give them – if the page renders content via JS, they will see only the shell. Use Firecrawl, Jina Reader, ScrapingBee, ZenRows, or run Playwright yourself to render first, then pipe the rendered HTML to a local extractor. Firecrawl and Crawl4AI bundle this into a single API call.

Extract Clean Markdown from Any URL: Tools and Approaches

A common AI-pipeline task: take a URL, return Markdown clean enough to feed an LLM. The naive answer ("just fetch the HTML") falls apart fast. Real web pages contain navigation chrome, ad slots, cookie banners, related-content widgets, comment sections, and deeply nested tracking divs that all look like content to a parser that doesn't know any better. Garbage in, garbage out.

This guide walks through the tools that actually solve this problem, when each one fits, and what trade-offs to expect.

URL-to-markdown vs. HTML-to-markdown: pick the right entry point

Most teams approach this as one problem; it is actually two. URL-to-markdown tools fetch and render the page for you – they handle JavaScript, anti-bot, and extraction in one call. HTML-to-markdown libraries take HTML you already have and clean it; they don't touch the network. Knowing which problem you have determines whether you reach for a hosted API or a local library, and the rest of this guide assumes you've made that distinction.

The four classes of tool

1. Hosted markdown-conversion APIs

These take a URL, return Markdown. You do not need to fetch the HTML yourself; the service handles rendering, anti-bot, and extraction in one call.

Jina Reader. Prepend https://r.jina.ai/ to any URL. The free tier handles low-volume use. The MCP version (read_url docs) gives Claude, GPT, and Gemini direct access for tool-use loops. Source is open (github.com/jina-ai/reader).

Firecrawl /scrape. A single API call returns Markdown plus optional structured fields. Firecrawl handles JS rendering, anti-bot, and post-processing into LLM-ready content (docs.firecrawl.dev). Pricing is per-page. Crawl mode handles whole-site ingestion (docs.firecrawl.dev/features/crawl). Extract mode pulls structured fields rather than free-form Markdown (docs.firecrawl.dev/features/extract).

Crawl4AI Cloud. Open-source under Apache 2.0 and the most popular AI-specific scraper on GitHub (docs.crawl4ai.com). Self-host for free; the cloud tier handles rendering and proxies. The headline feature is markdown output engineered specifically for LLMs – no boilerplate, no tracking junk.

When to pick a hosted API: when you need JS rendering, when target sites have anti-bot defenses, or when your team's time is more expensive than per-page fees.

2. Local extraction libraries

These take HTML you already fetched and return clean text or Markdown. They cannot fetch URLs themselves; that is a feature, not a bug – they are fast, deterministic, and free.

Trafilatura. Python library focused on main-content extraction. The ACL paper describes the algorithm; the docs cover practical usage. Trafilatura wins most accuracy benchmarks against open-source competitors. It is the right default for any Python pipeline that already has the HTML.

Mozilla Readability. The library Firefox uses for Reader Mode (github.com/mozilla/readability). Pure JavaScript, no dependencies. Better for Node pipelines or browser-side extraction. It is rule-based and fast, with quality that holds up surprisingly well against larger systems.

Beautiful Soup + custom rules. Not a markdown converter, but worth mentioning. If you have a small set of known sites with stable layouts, a 30-line Beautiful Soup extractor will outperform every general-purpose tool on those specific sites. The maintenance burden is the cost; it works as long as the sites don't redesign.

When to pick a local library: when you can fetch the HTML yourself, when latency matters, when per-page cost matters at scale, and when target sites don't require JS rendering or anti-bot bypass.

3. Agentic / LLM-driven extractors

These use an LLM to read the page and decide what counts as content. They handle long-tail page shapes that rule-based extractors miss.

Diffbot. Maintains a knowledge graph of trillions of facts. The Article API (diffbot.com/products/knowledge-graph) returns structured article objects with title, author, date, body, and clean text. It has been doing this since 2008 and remains the most accurate option for long-tail news and blog content.

AgentQL. GraphQL-style queries over a webpage. You write a schema describing what you want; the tool extracts it. Good fit when you need structured output from many similar pages but the layouts vary slightly.

Scrapegraph AI. Open-source agent framework specifically for scraping. You describe the data you want in natural language; the agent navigates and extracts.

When to pick an agentic extractor: when page layouts are heterogeneous, when you need structured output (not just Markdown), and when you can absorb the per-extraction LLM cost.

4. Browser-based scraping APIs that emit Markdown

Several scraping APIs produce Markdown as one of their output formats. These overlap with category 1 but are positioned as general scraping infrastructure with Markdown as a feature, not the headline.

ScrapingBee, ZenRows, Scrapfly, Bright Data Web Unlocker, Apify Actors, and ScraperAPI all return either rendered HTML or Markdown depending on configuration. They handle anti-bot, proxy rotation, and JS rendering as core features.

When to pick a scraping API: when you already use one for proxy and anti-bot handling and don't want to add a second vendor for Markdown extraction. Quality is good but generally trails the dedicated extractors above.

A decision tree

The fastest way to pick:

Are you prototyping? Use Jina Reader. Prepend https://r.jina.ai/ to your URL and ship.
Do you have the HTML already? Use Trafilatura (Python) or Readability (Node). Free, fast, deterministic.
Do target pages need JS rendering? Use Firecrawl, Crawl4AI Cloud, or Jina Reader. Skip the local libraries.
Do target pages have anti-bot defenses (Cloudflare, Akamai, DataDome)? Use Firecrawl, ScrapingBee, ZenRows, or Bright Data – the dedicated scraping APIs handle this; markdown libraries do not.
Do you need structured fields (price, author, date) and not just freeform Markdown? Use Firecrawl Extract, Diffbot, AgentQL, or Scrapegraph AI.
Are you doing whole-site ingestion? Use Firecrawl Crawl, Crawl4AI Cloud, or Apify – pure extractors don't handle link discovery.

Common failure modes

A few patterns that bite teams adopting these tools:

Boilerplate creeping in. Tools differ on how aggressively they strip nav, footer, sidebar. If your downstream embedding model is producing low-quality vectors, check whether the extracted Markdown still contains site-wide chrome. Trafilatura is conservative-by-default; Readability is aggressive. Firecrawl and Crawl4AI sit in between.

JS rendering inconsistency. "Renders JavaScript" is not a binary. Some tools render but stop early; others wait for a configurable selector. If your extractor returns blank content for known-good pages, the rendering step is the suspect.

Token budget blowup. Markdown of a long page is often 4,000–10,000 tokens. If you are feeding many pages into a single LLM call, batch carefully. Some pipelines summarize each page to ~500 tokens before downstream use; others chunk and embed.

Quality variance across domains. No single extractor wins everywhere. The Web Content Extraction Benchmark (webcontentextraction.org) is a useful methodology template – run a sample of your real target domains through two or three extractors, score against human-labeled output, pick the one that wins on your specific corpus.

For agentic-pipeline context on extraction quality more broadly, see our agentic web access guide. For the broader build-vs-buy economics around web data, see build vs buy.

Cost and latency, roughly

A rough calibration for production budgets in May 2026:

Local extractors (Trafilatura, Readability): essentially free. 10–100ms per page on commodity hardware.
Jina Reader free tier: rate-limited but free. Paid tier from $0.001/page.
Firecrawl: roughly $0.001–$0.005 per page depending on plan and rendering.
Crawl4AI self-hosted: free, plus your compute. Cloud tier roughly $0.002/page.
Diffbot: custom pricing; targets enterprise. Quality justifies the price for news/article corpora.
Bright Data Web Unlocker: $0.001–$0.005/page; the highest anti-bot success rate but no Markdown post-processing layer.

Latency for hosted APIs typically sits at 1–5 seconds per page. If your product needs sub-second extraction, the answer is usually local extractors over already-cached HTML, not faster APIs.

The right pattern for most AI products: extract once, cache the Markdown, re-embed only when the source page changes. Markdown extraction is one of the cheaper line items in an AI pipeline; treating it as a build-once-cache-forever problem is the configuration most teams converge on.