robots.txt for AI crawlers
Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.
Use separate robots.txt groups when an AI provider exposes separate search, training, and user-fetch identities. Start with the purpose you want to allow, write explicit rules for each named token, then verify the policy at the edge and origin.
What robots.txt can control
The Robots Exclusion Protocol lets a site publish crawl preferences at /robots.txt. It is a request to cooperating crawlers. The standard is explicit that these rules are not access authorization.
Do not put secrets behind a Disallow rule. The file is public, blocked paths are visible, and a client can ignore the policy. Use authentication, network controls, and application permissions for protected content.
Put the file on every host you mean to control
A robots.txt file applies to the scheme, hostname, and port where it is served. A rule at https://example.com/robots.txt does not automatically cover https://docs.example.com. Publish and test each public host separately.
The file should be available as UTF-8 plain text at the exact lowercase path:
https://example.com/robots.txtCheck the final response after redirects. A CDN, platform rule, or security product can serve a different file from the one in your repository.
Policy recipe: allow search and retrieval, block training
This example allows named search crawlers and expresses an allow preference for named user fetchers. It blocks named training controls:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Read the vendor notes before treating every line the same:
- OpenAI says
ChatGPT-Useris user initiated and robots.txt rules may not apply. - Anthropic documents separate controls for
ClaudeBot,Claude-SearchBot, andClaude-User. - Perplexity says
Perplexity-Usergenerally ignores robots.txt because the fetch is user requested. - Google says
Google-Extendedis a control token, not a separate HTTP user agent. Its setting does not affect Google Search inclusion or ranking.
Policy recipe: block named automatic AI crawlers
If the policy is to stop the named automatic search and training crawlers, make that intent direct:
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /This does not cover every crawler on the web. It also does not guarantee that user-triggered fetchers will treat the file as an authorization boundary.
Policy recipe: allow public docs, block operational paths
Path rules are useful when public documentation should remain crawlable but account, preview, or internal endpoints should not be crawled:
User-agent: OAI-SearchBot
Allow: /docs/
Allow: /blog/
Disallow: /account/
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
Allow: /docs/
Allow: /blog/
Disallow: /account/
Disallow: /admin/
Disallow: /api/Named groups are evaluated independently. Do not assume the rules under User-agent: * will also be combined with a matching named group. Put the paths that identity needs in its own group.
Know how path matching works
Under RFC 9309, matching starts at the beginning of the URI path. The most specific matching rule wins. Specificity is based on the length of the matched path. When an Allow and Disallow rule are equivalent, the allow rule should win.
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/In this group, /docs/private/plan is disallowed and /docs/public/api is allowed because the allow path is more specific. Path matching should be treated as case sensitive.
If the same product token appears in multiple groups, compliant crawlers combine the rules from those matching groups. Keep one group per identity when possible so a policy is easy to audit.
Test the public response
Do not stop at a repository diff. Test what a crawler can fetch:
curl -i https://example.com/robots.txtVerify these points:
- The final status is successful.
- The content type is plain text.
- The body contains the deployed rules.
- The same check is run for each subdomain.
- CDN caching does not leave an older policy in place.
- A parser returns the expected verdict for representative paths and user agents.
A missing file is not a safe default. RFC 9309 allows crawlers to access resources when robots.txt is unavailable due to a client-error response such as 404. Network and server failures have different handling. Keep the file small, stable, and reachable.
Check the WAF separately
A robots.txt allow does not override a WAF block. A crawler can be allowed by policy and still receive 403, 429, or a challenge page. Common causes include bot-score thresholds, stale IP allowlists, rate limits, geographic rules, and origin protections.
For vendors that publish IP ranges, verify both the user-agent claim and the current source range before creating a privileged WAF bypass. Store the vendor source and retrieval time with the rule. Do not rely on the header alone.
Observe the result
After deployment, inspect requests for the exact host and paths that matter. Record the claimed identity, verification state, HTTP outcome, and the robots verdict calculated from the file that was live at the time.
A clean policy with no crawler requests is still no request evidence. A crawler request with a 200 response is still not proof of indexing or citation. Keep policy, access, and downstream use as separate claims.
Use SearchSeal's public robots.txt checker for a quick policy read. For identity details, see the three OpenAI crawler identities and the cross-vendor user-agent reference.