Developer
Why ChatGPT and Claude Get 403 or 429 Errors
Diagnose 403 and 429 responses for ChatGPT and Claude crawlers across robots.txt, CDN, WAF, and application rules.
August 4, 20267 min read
Developer guides
AI crawler guides for developers
Identify AI crawlers, set access rules, and verify what reached your site.
IdentifyVerifyControlObserve
Developer
How to Track ChatGPT Crawlers on Vercel
Track ChatGPT, Claude, and other AI crawlers using Vercel Firewall events, AI bot rules, and request logs.
August 4, 20266 min read
Developer
How to Track ChatGPT Crawlers on WordPress
Find ChatGPT and Claude crawler requests to WordPress using CDN, host, or web-server access logs.
August 4, 20266 min read
Developer
How to Track ChatGPT Crawlers in AWS CloudFront
Use CloudFront request logs to find ChatGPT and Claude crawler paths, user agents, and HTTP outcomes.
August 4, 20267 min read
Developer
How to Track ChatGPT Crawlers in nginx Logs
Configure nginx access logs to record ChatGPT and Claude crawler paths, user agents, times, and HTTP statuses.
August 4, 20267 min read
Developer
GPTBot, OAI-SearchBot, and ChatGPT-User
How GPTBot, OAI-SearchBot, and ChatGPT-User differ, which robots.txt rules apply, and how to verify requests in server logs.
July 20, 20264 min read
Developer
robots.txt for AI crawlers
Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.
July 20, 20265 min read
Developer
AI crawler user agents: a developer reference
A developer reference for AI crawler user agents, robots.txt tokens, published IP ranges, purposes, and verification methods.
July 20, 20264 min read
Check your current crawler policy
Test a public robots.txt file against named AI crawlers.
Check robots.txt