Why ChatGPT and Claude Get 403 or 429 Errors
Diagnose 403 and 429 responses for ChatGPT and Claude crawlers across robots.txt, CDN, WAF, and application rules.
AI crawler guide for devs
Identify AI crawlers, set access rules, and verify what reached your site.
Choose a guide
Start with the infrastructure or crawler policy you need to inspect
How to Track ChatGPT Crawlers on Vercel
Track ChatGPT, Claude, and other AI crawlers using Vercel Firewall events, AI bot rules, and request logs.
How to Track ChatGPT Crawlers on WordPress
Find ChatGPT and Claude crawler requests to WordPress using CDN, host, or web-server access logs.
How to Track ChatGPT Crawlers in AWS CloudFront
Use CloudFront request logs to find ChatGPT and Claude crawler paths, user agents, and HTTP outcomes.
How to Track ChatGPT Crawlers in nginx Logs
Configure nginx access logs to record ChatGPT and Claude crawler paths, user agents, times, and HTTP statuses.
GPTBot, OAI-SearchBot, and ChatGPT-User
How GPTBot, OAI-SearchBot, and ChatGPT-User differ, which robots.txt rules apply, and how to verify requests in server logs.
robots.txt for AI crawlers
Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.
AI crawler user agents: a developer reference
A developer reference for AI crawler user agents, robots.txt tokens, published IP ranges, purposes, and verification methods.
Check your current crawler policy
Test a public robots.txt file against named AI crawlers.