Boilerplate Removal

Boilerplate removal is the step that isolates a page's main content from everything wrapped around it: site navigation, ads, cookie banners, newsletter interstitials, related-post rails, share widgets, comment sections and footers. It is the unglamorous stage that decides whether LLM-ready output is usable at all, because template text that survives extraction is repeated on every page of a site and therefore appears in every chunk you index from that site.

Three approaches are in production use, usually in combination. Readability heuristics score DOM nodes on signals like text length, comma count, tag type and telltale class or id names, then keep the highest-scoring subtree; Mozilla Readability, the library behind Firefox Reader Mode, is the reference implementation and is embedded in much of the downstream tooling. Text density scoring compares the ratio of text to markup across candidate subtrees, on the reasoning that a nav bar is many links and few words while an article body is the reverse. Machine-learned classifiers score nodes on a broader feature set and generalise better across unfamiliar templates at the cost of being harder to debug when they get something wrong. Trafilatura, widely used in building LLM training corpora, layers rules and fallbacks rather than betting on any single method.

Two failure modes matter, and they pull in opposite directions. Over-stripping removes content that structurally resembles furniture: pricing tables, specification lists, code blocks, footnotes and figure captions are the usual casualties, and they are frequently the exact rows a user is asking about. Under-stripping leaves template text in place, which is worse than it looks. Boilerplate that survives is identical across thousands of documents, so it inflates token cost on every retrieval, and in an embedding index it pushes chunks from the same site toward each other on the basis of the shared navigation rather than the content, degrading retrieval precision for the whole corpus.

Both are cheap to test. Extract two unrelated pages from the same domain and diff the outputs: anything identical is boilerplate that survived. Then take the pages in your corpus that carry the actual answer in a table or a code block and check those specific elements survived the round trip. Run this on your own sources rather than on a vendor's demo URL, because extraction quality is a function of the templates you happen to hit, and a tool that is excellent on news articles can be poor on documentation sites or e-commerce pages.

The buy-or-build line here is a maintenance question rather than a capability one. Trafilatura and Mozilla Readability are open source, free, and good enough for a large share of the open web, but you own the failures when a major source redesigns. Firecrawl and Jina AI's Reader API sell extraction as a call and absorb that maintenance, along with JavaScript rendering and anti-bot handling, which are usually the reason teams stop self-hosting. If your corpus is a fixed set of sources you control, self-hosting is defensible. If it is the open web, someone has to keep the parsers current, and it is worth being honest about whether that will be you.