User Agent

A user agent is the string a client sends in the `User-Agent` HTTP header to identify itself. Browsers send strings like `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36`. Bots and scripts send anything from `python-requests/2.31` (obviously a bot) to a spoofed Chrome string (an attempt to appear as a browser). Servers use the user agent both for legitimate purposes (delivering mobile-optimized content, recording analytics) and for filtering (blocking known automation libraries by default). For scraping, setting a realistic user agent is the lowest-effort first step toward avoiding blocks. The default user agents of `requests`, `httpx`, and similar libraries are explicitly blacklisted by many sites; even rotating to a random Chrome string improves baseline success rates. More sophisticated systems track user agent in combination with other signals (TLS fingerprint, header order, JavaScript challenge solutions), so spoofing the user agent alone is insufficient against serious anti-bot stacks. For AI builders, the practical advice is to keep a small library of recent, real browser user agents (refresh every quarter as new browser versions ship) and to rotate them per request. Make sure the rest of the request looks consistent with the chosen user agent — a Chrome user agent paired with Python's default Accept-Language header is itself a tell. Most scraping APIs handle user agent rotation and consistency automatically.