robots.txt for AI crawlers
Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.
Use separate robots.txt groups when an AI provider exposes separate search, training, and user-fetch identities. Start with the purpose you want to allow, write explicit rules for each named token, then verify the policy at the edge and origin.
What robots.txt can control
The Robots Exclusion Protocol lets a site publish crawl preferences at /robots.txt. It is a request to cooperating crawlers. The standard is explicit that these rules are not access authorization.
Do not put secrets behind a Disallow rule. The file is public, blocked paths are visible, and a client can ignore the policy. Use authentication, network controls, and application permissions for protected content.
Put the file on every host you mean to control
A robots.txt file applies to the scheme, hostname, and port where it is served. A rule at https://example.com/robots.txt does not automatically cover https://docs.example.com. Publish and test each public host separately.
The file should be available as UTF-8 plain text at the exact lowercase path:
https://example.com/robots.txtCheck the final response after redirects. A CDN, platform rule, or security product can serve a different file from the one in your repository.
Policy recipe: allow search and retrieval, block training
This example allows named search crawlers and expresses an allow preference for named user fetchers. It blocks named training controls:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Read the vendor notes before treating every line the same:
- OpenAI says
ChatGPT-Useris user initiated and robots.txt rules may not apply. - Anthropic documents separate controls for
ClaudeBot,Claude-SearchBot, andClaude-User. - Perplexity says
Perplexity-Usergenerally ignores robots.txt because the fetch is user requested. - Google says
Google-Extendedis a control token, not a separate HTTP user agent. Its setting does not affect Google Search inclusion or ranking.
Policy recipe: block named automatic AI crawlers
If the policy is to stop the named automatic search and training crawlers, make that intent direct:
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /This does not cover every crawler on the web. It also does not guarantee that user-triggered fetchers will treat the file as an authorization boundary.
Policy recipe: allow public docs, block operational paths
Path rules are useful when public documentation should remain crawlable but account, preview, or internal endpoints should not be crawled:
User-agent: OAI-SearchBot
Allow: /docs/
Allow: /blog/
Disallow: /account/
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
Allow: /docs/
Allow: /blog/
Disallow: /account/
Disallow: /admin/
Disallow: /api/Named groups are evaluated independently. Do not assume the rules under User-agent: * will also be combined with a matching named group. Put the paths that identity needs in its own group.
Know how path matching works
Under RFC 9309, matching starts at the beginning of the URI path. The most specific matching rule wins. Specificity is based on the length of the matched path. When an Allow and Disallow rule are equivalent, the allow rule should win.
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/In this group, /docs/private/plan is disallowed and /docs/public/api is allowed because the allow path is more specific. Path matching should be treated as case sensitive.
If the same product token appears in multiple groups, compliant crawlers combine the rules from those matching groups. Keep one group per identity when possible so a policy is easy to audit.
Test the public response
Do not stop at a repository diff. Test what a crawler can fetch:
curl -i https://example.com/robots.txtVerify these points:
- The final status is successful.
- The content type is plain text.
- The body contains the deployed rules.
- The same check is run for each subdomain.
- CDN caching does not leave an older policy in place.
- A parser returns the expected verdict for representative paths and user agents.
A missing file is not a safe default. RFC 9309 allows crawlers to access resources when robots.txt is unavailable due to a client-error response such as 404. Network and server failures have different handling. Keep the file small, stable, and reachable.
Check the WAF separately
A robots.txt allow does not override a WAF block. A crawler can be allowed by policy and still receive 403, 429, or a challenge page. Common causes include bot-score thresholds, stale IP allowlists, rate limits, geographic rules, and origin protections.
For vendors that publish IP ranges, verify both the user-agent claim and the current source range before creating a privileged WAF bypass. Store the vendor source and retrieval time with the rule. Do not rely on the header alone.
Observe the result
After deployment, inspect requests for the exact host and paths that matter. Record the claimed identity, verification state, HTTP outcome, and the robots verdict calculated from the file that was live at the time.
A clean policy with no crawler requests is still no request evidence. A crawler request with a 200 response is still not proof of indexing or citation. Keep policy, access, and downstream use as separate claims.
For identity details, see the three OpenAI crawler identities and the cross-vendor user-agent reference.
Allow GPTBot but block other AI bots
To allow GPTBot but block other AI bots, give GPTBot its own Allow: / group and give each unwanted bot a Disallow: / group. This example allows GPTBot and blocks the named agents; it is not an exhaustive list of AI crawlers.
Updated October 2026.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /A User-agent: * rule applies to bots with no matching group of their own. A specific matching group overrides that wildcard group, per the robots exclusion standard. Preserve any path restrictions you still need inside the specific group; wildcard restrictions are not inherited.
Google-Extended is a control token rather than a separate crawler, per Google's documentation. Its rule controls specified AI uses of crawled content without blocking Google Search.
Per OpenAI's documentation, GPTBot collects content potentially used for training; OAI-SearchBot handles search. Allowing GPTBot does not guarantee a citation. Review these separate OpenAI crawler roles, merge the example with existing rules, and check the result with the AI crawler checker. Robots.txt expresses access preferences; it does not enforce firewall blocks.
Does GPTBot support crawl-delay in robots.txt?
OpenAI's documentation lists no crawl-delay support for GPTBot, so do not rely on it to control request volume. For a GPTBot crawl delay, enforce rate limits at your server or CDN and use appropriate 429 or 503 responses when requests exceed capacity.
Updated October 2026.
Per OpenAI's documentation, GPTBot has a documented user agent and published IP ranges. Verify requests against those ranges before applying a crawler-specific policy; a user-agent string alone can be copied. Choose limits from your site's capacity and observed load, rather than assuming a universal safe interval.
HTTP 429 means too many requests. HTTP 503 signals temporary unavailability, such as overload or maintenance. A Retry-After header can communicate when to try again, but neither status establishes a documented GPTBot retry schedule. Keep the limit enforced on your infrastructure.
Use server or CDN logs to compare request counts and response codes after the change. The AI crawler checker checks robots.txt access rules, not live request rates. Keep that access check separate from load monitoring, and consult the OpenAI crawler guide before applying one policy to different agents.
How to unblock GPTBot after blocking it
To unblock GPTBot after blocking it, change its robots.txt group to Allow: / and remove conflicting GPTBot restrictions. Also check CDN, WAF and hosting-level blocks: an allow rule in robots.txt cannot override them.
Updated October 2026.
User-agent: GPTBot
Allow: /You can instead remove the blocking GPTBot group if the fallback rules permit access. If User-agent: * still disallows the site, removing the specific group leaves GPTBot blocked by that fallback. Review all matching GPTBot groups and any restrictions on the paths you want crawled.
Fetch /robots.txt from the live hostname and confirm that the response contains the revised rules. Check for an old CDN-cached copy, a redirect to another hostname, or a hosting-generated file that differs from the one you edited. Then verify the published rules with the AI crawler checker.
Inspect security rules and server logs separately. Per OpenAI's documentation, GPTBot's published IP ranges help identify its requests. A checker result does not prove that the firewall permits those requests.
Crawl changes are not instant, and there is no guaranteed GPTBot recrawl time. Allowing access does not schedule a visit or guarantee inclusion in an answer. The OpenAI crawler guide explains why training access and search access are separate decisions.