HTML Parser

An HTML parser is a library that turns raw HTML text into a navigable tree structure – a DOM or DOM-like object – so your code can query elements, extract text, and read attributes. The core challenge is that real-world HTML is often broken: unclosed tags, missing quotes, inconsistent nesting, and legacy quirks. Browsers handle this with permissive parsing rules defined by the HTML5 specification; libraries like Beautiful Soup (Python), Cheerio (Node), and lxml implement those rules so your scraper can work on the same kind of malformed HTML browsers can. For scraping, the parser is paired with a query mechanism – usually CSS selectors (`#main .product-title`) or XPath expressions (`//div[@class='product']/h2`). CSS selectors are easier to read and write; XPath is more powerful when you need to navigate up the tree, match by text content, or apply complex predicates. Most modern scraping libraries support both. For AI builders, the practical choice between parsers is rarely strategic – pick the one that fits your language. The strategic question is whether to parse with selectors at all. LLM-based extraction (sending the HTML or a cleaned text version to an LLM with a target schema) is more robust against site changes and easier to set up, but it costs orders of magnitude more per page. For high-volume scraping with stable targets, selectors win; for low-volume scraping with messy or shifting targets, LLM extraction often wins.