Developer
GPTBot, OAI-SearchBot, and ChatGPT-User
How GPTBot, OAI-SearchBot, and ChatGPT-User differ, which robots.txt rules apply, and how to verify requests in server logs.
July 20, 20264 min read
Developer guides
AI crawler guides for developers
Identify AI crawlers, set access rules, and verify what reached your site.
IdentifyVerifyControlObserve
Developer
robots.txt for AI crawlers
Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.
July 20, 20265 min read
Developer
AI crawler user agents: a developer reference
A developer reference for AI crawler user agents, robots.txt tokens, published IP ranges, purposes, and verification methods.
July 20, 20264 min read
Check your current crawler policy
Test a public robots.txt file against named AI crawlers.
Check robots.txt