AI bots & crawlers

Classification & verification

How bot hits are classified, how spoofed crawlers are caught, and what leaves your server.

Classification

Every hit is matched to a product (OpenAI, Anthropic, Perplexity, Google…) and a category:

CategoryExampleWhat it means
Training crawlerGPTBot, ClaudeBotCollects pages to train models
AI search indexOAI-SearchBot, PerplexityBotBuilds an index for AI search answers
Live fetch for a userChatGPT-User, Perplexity-UserFetches a page because a user asked about it
AgentChatGPT agent, Google-AgentActs on the site for a user

Verification

Anyone can send User-Agent: GPTBot. Each hit is checked against these signals, strongest first:

  1. Web Bot Auth: a cryptographic request signature from the vendor
  2. CDN verdict: Cloudflare's verified-bot flag, from log drains
  3. Published IP ranges: the vendor's official list, refreshed daily
  4. Reverse DNS: the IP resolves to the vendor's domain and back

Hits that claim to be a known bot but fail verification are flagged as spoofed and left out of the numbers. Hits that can't be checked either way, such as those with an internal IP, are shown as unverified.

Agents that use a real browser

Some agents, such as ChatGPT agent and Google-Agent, browse with a real browser. For those, the server SDK sets a short-lived _al_v cookie. The browser SDK reads it, so the WebMCP tool calls that agent then makes are attributed to it. The cookie is never set for humans. Turn it off with visitCookie: false.

What leaves your server

  • Bot requests: method, URL, status, user agent, signature headers and IP. The IP is used for verification and then discarded.
  • AI-referred landing pages: path and referrer host, for visits whose referrer is an AI assistant. The query string is dropped.
  • Human page views: a single count per day.

Nothing else about humans is sent.