Agent Harness
An agent harness is the software wrapped around a language model that actually runs the agent loop: dispatching tool calls, persisting…
Glossary
Key terms in AI search, web data, and the infrastructure that powers LLMs and AI agents.
Agent Harness
Agentic Extraction
Agentic extraction is an approach to web data collection where an AI model actively navigates, interprets, and extracts information from…
Agentic Search
Agentic search is a retrieval strategy in which a model issues a search, reads the results, reformulates the query, and decides for…
Agentic Web Extraction
Agentic web extraction is the broad category of web data collection where an AI agent – not a hand-written scraper – decides what to…
AI Agent
An AI agent is a software system built around a language model that can autonomously plan, execute multi-step tasks, and interact with…
AI Crawler
An AI crawler is an automated client operated by a model provider or AI search company that fetches web pages for use in an AI system…
AI Data Pipeline
An AI data pipeline is the end-to-end system that collects, processes, and delivers external data to an AI application.
AI Overview
AI Overview is Google's name for the AI-generated summary that appears at the top of certain search result pages.
AI Search API
An AI search API is a web service that lets applications query the internet and receive results optimized for consumption by large…
Answer Engine Optimization (AEO)
Answer Engine Optimization, or AEO, is the practice of optimizing for presence and accuracy inside AI-generated answers rather than for…
Anti-Bot Detection
Anti-bot detection is the layer of defenses websites use to identify and block automated traffic.
API Credits
API credits are the abstract billing unit that most AI search and web extraction vendors meter usage in: you buy or are allotted a pool…
Benchmark Saturation
Benchmark saturation is the point at which a benchmark stops separating systems, because the leading entrants all sit close to its…
Boilerplate Removal
Boilerplate removal is the step that isolates a page's main content from everything wrapped around it: site navigation, ads, cookie…
BrowseComp
BrowseComp is a benchmark of 1,266 short-answer questions, open-sourced by OpenAI in April 2025, that measures whether an agent can find…
Browser Fingerprinting
Browser fingerprinting is the technique of identifying a browser by combining many small, individually unremarkable signals into a…
CAPTCHA
CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart.
Chrome DevTools Protocol (CDP)
The Chrome DevTools Protocol is the JSON-over-WebSocket interface that Chromium exposes for instrumenting and controlling a browser from…
Chunking
Chunking is the step in a retrieval pipeline where a document is split into smaller units that are embedded, indexed, and retrieved…
Client-Side Rendering (CSR)
Client-side rendering is the pattern where the server ships a near-empty HTML shell and the browser's JavaScript renders the actual page…
Cloud Browser Session
A cloud browser session is a single isolated browser instance provisioned on a vendor's infrastructure and driven remotely over the…
Computer Use
Computer use is a form of automation in which a model operates a graphical interface the way a person does: it receives screenshots…
Content Signals Policy
The Content Signals Policy is Cloudflare's extension to robots.txt that lets a site owner state, in a machine-readable line, how a…
Context Engineering
Context engineering is the practice of deciding what information enters a model's context window, in what form, and in what order, for a…
Context Rot
Context rot is the degradation in an LLM's retrieval accuracy and instruction-following as its context window fills, even when the…
Context Window
The context window is the maximum amount of text – measured in tokens – that a language model can process in a single request, including…
Crawl Budget
Crawl budget is the term for the maximum number of pages a search engine or other crawler will fetch from a given site within a time window.
CSS Selector
A CSS selector is a string syntax for matching elements in an HTML document, originally designed for stylesheets but widely used for DOM…
Data Freshness
Data freshness refers to how current the information is that an AI system has access to when generating responses.
Datacenter Proxy
A datacenter proxy is an IP address allocated to a server in a commercial cloud or hosting provider.
Deep Research Agent
A deep research agent is an agent that plans a multi-step investigation, issues many searches, reads the sources it finds, and returns a…
Embeddings
Embeddings are numerical vector representations of text (or images, audio, and other data) produced by neural networks.
Generative Engine Optimization (GEO)
Generative Engine Optimization, or GEO, is the practice of optimizing content to be cited and synthesized by AI search and AI Overview…
Grounding
Grounding is the practice of anchoring a language model's output to verifiable external sources.
Hallucination Prevention
Hallucination prevention encompasses the techniques and system design patterns used to reduce the rate at which AI models generate…
Headless Browser
A headless browser is a real web browser – usually Chromium, Firefox, or WebKit – running without a graphical user interface.
Honeypot Trap
A honeypot trap is a hidden element placed on a web page specifically to catch automated clients.
HTML Parser
An HTML parser is a library that turns raw HTML text into a navigable tree structure – a DOM or DOM-like object – so your code can query…
Hybrid Search
Hybrid search runs a lexical retriever and a dense vector retriever over the same corpus and merges their two ranked lists into one.
JavaScript Rendering
JavaScript rendering, often abbreviated JS rendering, refers to executing the JavaScript on a web page so the resulting DOM contains the…
Knowledge Cutoff
A knowledge cutoff is the date after which a language model's training data ends.
Live Crawling
Live crawling is fetching a web page at the moment a query is made, rather than returning a copy that was crawled and stored earlier.
LLM-Ready Data
LLM-ready data is web content that has been cleaned, structured, and formatted for direct consumption by a large language model.
LLM-Ready Markdown
LLM-ready Markdown is the specific output format that modern scraping APIs produce for AI pipelines: a single Markdown document per…
llms.txt
The llms.txt file is a proposed standard for sites to publish a Markdown-formatted summary of their most important content for…
MCP (Model Context Protocol)
The Model Context Protocol, or MCP, is an open standard introduced by Anthropic in late 2024 that defines how AI models connect to…
Mobile Proxy
A mobile proxy routes requests through a real mobile carrier IP – typically a 4G or 5G connection on a phone or USB modem.
Multi-Hop Retrieval
Multi-hop retrieval is answering a question that requires chaining evidence across two or more documents, where the result of each…
Online-Mind2Web
Online-Mind2Web is the live-web extension of the Mind2Web benchmark, introduced to evaluate web agents on real, public websites rather…
OSWorld
OSWorld is a benchmark for computer-use agents, released in 2024 by researchers from the University of Hong Kong, Salesforce Research…
Prompt Injection
Prompt injection is an attack in which text that a language model reads as input is treated by the model as instructions to obey.
Proxy Pool
A proxy pool is a managed collection of proxy IP addresses that a scraper or scraping platform draws from to spread requests.
Query Fan-Out
Query fan-out is the technique of decomposing a single user question into many sub-queries, issuing them in parallel, and synthesising…
RAG Evaluation
RAG evaluation is the practice of scoring a retrieval-augmented generation system with metrics that separate retrieval quality from…
Rate Limiting
Rate limiting is the practice of capping how many requests a client can make to a server within a time window.
Real-Time Web Access
Real-time web access is the capability of an AI system to retrieve current information from the internet at the moment a user asks a…
Remote MCP Server
A remote MCP server is a Model Context Protocol server that a client reaches over HTTP at a URL, hosted and operated by the provider…
Reranking
Reranking is a second-pass scoring step that takes the candidate documents returned by an initial retrieval and reorders them with a…
Residential Proxy
A residential proxy is an IP address assigned by a consumer internet service provider to a real home or mobile device, then routed…
Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation, or RAG, is an architecture pattern where a language model's response is informed by external documents…
robots.txt
The robots.txt file is a plain-text file at the root of a web domain that declares which paths automated agents are allowed or…
Rotating Proxy
A rotating proxy is a proxy service that automatically swaps the outbound IP address on every request, or on a configured time interval.
Schema-Guided Extraction
Schema-guided extraction is the practice of passing a target schema, typically JSON Schema or a Pydantic model, to a language model…
Search-Augmented Generation
Search-augmented generation is a specific form of RAG where the retrieval step queries a live search engine rather than a static…
Self-Healing Selectors
Self-healing selectors are extraction targets that keep working after a site changes its markup, because the locator is stored as a set…
Semantic Search
Semantic search is a retrieval method that finds results based on the meaning of a query rather than exact keyword matches.
SERP API
A SERP API is a service that programmatically retrieves search engine results pages and returns the data in a structured format –…
SERP Feature
A SERP feature is any block on a search engine results page that is not one of the standard organic blue links.
Server-Side Rendering (SSR)
Server-side rendering is the pattern where the server generates the full HTML for a page before sending it to the client.
Stealth Mode
Stealth mode, in the context of web automation, refers to a set of patches applied to headless browsers to hide the fact that they are…
Structured Output
Structured output refers to an LLM's ability to generate responses in a specific, machine-readable format – typically JSON, but also…
TLS Fingerprinting
TLS fingerprinting identifies a client by the precise pattern of cipher suites, extensions, and elliptic curves it advertises in its TLS…
Tool Use
Tool use, also called function calling, is the ability of a language model to invoke external functions or APIs as part of generating a…
User Agent
A user agent is the string a client sends in the `User-Agent` HTTP header to identify itself.
Vector Database
A vector database is a storage and retrieval system optimized for high-dimensional numeric vectors – the embeddings produced by models…
Vector Search
Vector search is the retrieval technique that finds the most semantically similar items to a query by comparing high-dimensional…
Vision-Based Extraction
Vision-based extraction is pulling structured data out of a rendered screenshot with a multimodal model, rather than parsing the page's DOM.
Web Bot Auth
Web Bot Auth is a scheme for cryptographic bot identity: an automated client signs its outbound HTTP requests with a private key, and…
Web Browsing Agent
A web browsing agent is an AI system that can autonomously navigate, interact with, and extract information from websites using a real…
Web Crawler
A web crawler – sometimes called a spider or bot – is a program that systematically discovers URLs and downloads pages.
Web Data Extractor
A web data extractor (often called a web scraper, the legacy term) is a program that extracts specific data from web pages.
Web Index
A web index is a searchable database of web pages that has been built by systematically crawling and processing the internet.
WebArena
WebArena is an academic benchmark released in 2023 by researchers at Carnegie Mellon University for evaluating autonomous web agents on…
XML Sitemap
An XML sitemap is a structured file – typically at /sitemap.xml – that lists the URLs a site wants search engines and crawlers to…
XPath
XPath is a query language for navigating XML and HTML documents.