Agentic Extraction
Agentic extraction is an approach to web data collection where an AI model actively navigates, interprets, and extracts information from…
Glossary
Glossary
Key terms in AI search, web data, and the infrastructure that powers LLMs and AI agents.
Agentic Extraction
Agentic Web Extraction
Agentic web extraction is the broad category of web data collection where an AI agent – not a hand-written scraper – decides what to…
AI Agent
An AI agent is a software system built around a language model that can autonomously plan, execute multi-step tasks, and interact with…
AI Data Pipeline
An AI data pipeline is the end-to-end system that collects, processes, and delivers external data to an AI application.
AI Overview
AI Overview is Google's name for the AI-generated summary that appears at the top of certain search result pages.
AI Search API
An AI search API is a web service that lets applications query the internet and receive results optimized for consumption by large…
Anti-Bot Detection
Anti-bot detection is the layer of defenses websites use to identify and block automated traffic.
Browser Fingerprinting
Browser fingerprinting is the technique of identifying a browser by combining many small, individually unremarkable signals into a…
CAPTCHA
CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart.
Client-Side Rendering (CSR)
Client-side rendering is the pattern where the server ships a near-empty HTML shell and the browser's JavaScript renders the actual page…
Context Window
The context window is the maximum amount of text – measured in tokens – that a language model can process in a single request, including…
Crawl Budget
Crawl budget is the term for the maximum number of pages a search engine or other crawler will fetch from a given site within a time window.
CSS Selector
A CSS selector is a string syntax for matching elements in an HTML document, originally designed for stylesheets but widely used for DOM…
Data Freshness
Data freshness refers to how current the information is that an AI system has access to when generating responses.
Datacenter Proxy
A datacenter proxy is an IP address allocated to a server in a commercial cloud or hosting provider.
Embeddings
Embeddings are numerical vector representations of text (or images, audio, and other data) produced by neural networks.
Generative Engine Optimization (GEO)
Generative Engine Optimization, or GEO, is the practice of optimizing content to be cited and synthesized by AI search and AI Overview…
Grounding
Grounding is the practice of anchoring a language model's output to verifiable external sources.
Hallucination Prevention
Hallucination prevention encompasses the techniques and system design patterns used to reduce the rate at which AI models generate…
Headless Browser
A headless browser is a real web browser – usually Chromium, Firefox, or WebKit – running without a graphical user interface.
Honeypot Trap
A honeypot trap is a hidden element placed on a web page specifically to catch automated clients.
HTML Parser
An HTML parser is a library that turns raw HTML text into a navigable tree structure – a DOM or DOM-like object – so your code can query…
JavaScript Rendering
JavaScript rendering, often abbreviated JS rendering, refers to executing the JavaScript on a web page so the resulting DOM contains the…
LLM-Ready Data
LLM-ready data is web content that has been cleaned, structured, and formatted for direct consumption by a large language model.
LLM-Ready Markdown
LLM-ready Markdown is the specific output format that modern scraping APIs produce for AI pipelines: a single Markdown document per…
llms.txt
The llms.txt file is a proposed standard for sites to publish a Markdown-formatted summary of their most important content for…
MCP (Model Context Protocol)
The Model Context Protocol, or MCP, is an open standard introduced by Anthropic in late 2024 that defines how AI models connect to…
Mobile Proxy
A mobile proxy routes requests through a real mobile carrier IP – typically a 4G or 5G connection on a phone or USB modem.
Online-Mind2Web
Online-Mind2Web is the live-web extension of the Mind2Web benchmark, introduced to evaluate web agents on real, public websites rather…
OSWorld
OSWorld is a benchmark for computer-use agents, released in 2024 by researchers from the University of Hong Kong, Salesforce Research…
Proxy Pool
A proxy pool is a managed collection of proxy IP addresses that a scraper or scraping platform draws from to spread requests.
Rate Limiting
Rate limiting is the practice of capping how many requests a client can make to a server within a time window.
Real-Time Web Access
Real-time web access is the capability of an AI system to retrieve current information from the internet at the moment a user asks a…
Residential Proxy
A residential proxy is an IP address assigned by a consumer internet service provider to a real home or mobile device, then routed…
Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation, or RAG, is an architecture pattern where a language model's response is informed by external documents…
robots.txt
The robots.txt file is a plain-text file at the root of a web domain that declares which paths automated agents are allowed or…
Rotating Proxy
A rotating proxy is a proxy service that automatically swaps the outbound IP address on every request, or on a configured time interval.
Search-Augmented Generation
Search-augmented generation is a specific form of RAG where the retrieval step queries a live search engine rather than a static…
Semantic Search
Semantic search is a retrieval method that finds results based on the meaning of a query rather than exact keyword matches.
SERP API
A SERP API is a service that programmatically retrieves search engine results pages and returns the data in a structured format –…
Server-Side Rendering (SSR)
Server-side rendering is the pattern where the server generates the full HTML for a page before sending it to the client.
Stealth Mode
Stealth mode, in the context of web automation, refers to a set of patches applied to headless browsers to hide the fact that they are…
Structured Output
Structured output refers to an LLM's ability to generate responses in a specific, machine-readable format – typically JSON, but also…
TLS Fingerprinting
TLS fingerprinting identifies a client by the precise pattern of cipher suites, extensions, and elliptic curves it advertises in its TLS…
Tool Use
Tool use, also called function calling, is the ability of a language model to invoke external functions or APIs as part of generating a…
User Agent
A user agent is the string a client sends in the `User-Agent` HTTP header to identify itself.
Vector Database
A vector database is a storage and retrieval system optimized for high-dimensional numeric vectors – the embeddings produced by models…
Vector Search
Vector search is the retrieval technique that finds the most semantically similar items to a query by comparing high-dimensional…
Web Browsing Agent
A web browsing agent is an AI system that can autonomously navigate, interact with, and extract information from websites using a real…
Web Crawler
A web crawler – sometimes called a spider or bot – is a program that systematically discovers URLs and downloads pages.
Web Data Extractor
A web data extractor (often called a web scraper, the legacy term) is a program that extracts specific data from web pages.
Web Index
A web index is a searchable database of web pages that has been built by systematically crawling and processing the internet.
WebArena
WebArena is an academic benchmark released in 2023 by researchers at Carnegie Mellon University for evaluating autonomous web agents on…
XML Sitemap
An XML sitemap is a structured file – typically at /sitemap.xml – that lists the URLs a site wants search engines and crawlers to…
XPath
XPath is a query language for navigating XML and HTML documents.