AI crawler user agents: a developer reference

An AI crawler identity starts as a product token in an HTTP user-agent string or robots.txt group. It is not verified identity by itself. Use the token to classify a claim, then use vendor-published network ranges and request evidence where they are available.

Reference reviewed 20 July 2026. Vendor names, versions, purposes, and network ranges can change. The linked vendor documents are the source of truth.

OpenAI

Product token	Purpose	robots.txt note	Network source
OAI-SearchBot	Automatic crawl for ChatGPT Search	Independent named control	searchbot.json
GPTBot	Content that may be used for foundation-model training	Independent named control	gptbot.json
ChatGPT-User	Some user-initiated page fetches	Rules may not apply	chatgpt-user.json

OpenAI's official crawler reference publishes example full user-agent strings. The version number may change, so classify on the stable token. Use OAI-SearchBot for Search policy and GPTBot for training policy. ChatGPT-User is not the Search opt-out.

Anthropic

Product token	Purpose	robots.txt note	Network source
ClaudeBot	Content that may contribute to model training	Named control	bots.json
Claude-SearchBot	Search indexing and result quality	Named control	bots.json
Claude-User	User-directed web retrieval	Named control	bots.json

Anthropic documents all three identities in its crawler policy. Anthropic says its bots honor robots.txt directives and publishes one current network feed for crawler verification.

Perplexity

Product token	Purpose	robots.txt note	Network source
PerplexityBot	Automatic crawl for Perplexity search results	Named control	perplexitybot.json
Perplexity-User	User-requested page fetch	Generally ignores robots.txt	perplexity-user.json

Perplexity's crawler documentation states that PerplexityBot is not used to collect content for foundation-model training. It also recommends combining user-agent matching with its current published IP ranges for WAF rules.

Google-Extended is a control token, not a request identity

Google-Extended belongs in a robots.txt reference because publishers can use it to control certain Gemini training and grounding uses. It does not have a separate HTTP user-agent string. Google says the fetch is made with existing Google user agents.

User-agent: Google-Extended
Disallow: /

Do not search access logs for a fictional Google-Extended request and conclude there was no relevant activity. It is a policy token. Google also states that this control does not affect inclusion or ranking in Google Search. See Google's crawler reference.

Match stable tokens, not full versions

Full user-agent strings often contain browser compatibility text and version numbers. A literal equality check will break when a vendor updates either one. Start with a case-insensitive token match:

const patterns = [
  { token: 'OAI-SearchBot', vendor: 'OpenAI', purpose: 'search' },
  { token: 'GPTBot', vendor: 'OpenAI', purpose: 'training' },
  { token: 'ChatGPT-User', vendor: 'OpenAI', purpose: 'user_fetch' },
  { token: 'Claude-SearchBot', vendor: 'Anthropic', purpose: 'search' },
  { token: 'ClaudeBot', vendor: 'Anthropic', purpose: 'training' },
  { token: 'Claude-User', vendor: 'Anthropic', purpose: 'user_fetch' },
  { token: 'PerplexityBot', vendor: 'Perplexity', purpose: 'search' },
  { token: 'Perplexity-User', vendor: 'Perplexity', purpose: 'user_fetch' },
];

function identifyClaimedAgent(userAgent: string) {
  const normalized = userAgent.toLowerCase();
  return patterns.find(({ token }) =>
    normalized.includes(token.toLowerCase())
  ) ?? null;
}

Name the result claimedAgent, not verifiedAgent. That label prevents a user-agent substring from quietly becoming stronger evidence than it is.

Verify published ranges separately

Where a provider publishes network ranges, the verification flow is:

1. Identify the claimed product token from the complete header.
2. Fetch the current range document from the provider's HTTPS endpoint.
3. Parse both IPv4 and IPv6 prefixes when the feed supplies them.
4. Check the connection source IP against the appropriate prefixes.
5. Record the source URL, feed retrieval time, and verification result.

Preserve the actual peer address before trusting proxy headers. If traffic reaches your application through a CDN or load balancer, define which proxy sets the client IP and which forwarding header is authoritative. An untrusted X-Forwarded-For value is another requester-supplied claim.

Store evidence that can be reclassified

A useful crawler event keeps raw observation separate from interpretation:

{
  "observed_at": "2026-07-20T08:15:30Z",
  "host": "docs.example.com",
  "method": "GET",
  "path": "/guides/robots.txt",
  "status": 200,
  "user_agent": "...OAI-SearchBot/1.4...",
  "claimed_agent": "OAI-SearchBot",
  "network_verified": true,
  "robots_verdict": "allowed"
}

The raw user agent, source evidence, and policy version let you correct a classifier without rewriting history. Keep unknown as a valid result when the source does not expose enough evidence.

Do not collapse purpose into vendor

A vendor name is not a crawl purpose. OpenAI, Anthropic, and Perplexity each document distinct identities for different jobs. A report labeled only “OpenAI traffic” cannot answer whether the request was automatic search, training crawl, or user-triggered retrieval.

Keep these fields separate:

• Vendor: the organization publishing the identity.
• Product token: the crawler or fetcher name.
• Purpose: search, training, user fetch, or another documented job.
• Verification: claimed, network verified, or unknown.
• Outcome: the HTTP result observed by your infrastructure.

For copyable policy groups, read robots.txt for AI crawlers. For the OpenAI-specific split, read GPTBot, OAI-SearchBot, and ChatGPT-User.