AI crawler guide for devs

Identify AI crawlers, set access rules, and verify what reached your site.

Why ChatGPT and Claude Get 403 or 429 Errors

Diagnose 403 and 429 responses for ChatGPT and Claude crawlers across robots.txt, CDN, WAF, and application rules.

Track ChatGPT, Claude, and other AI crawlers using Vercel Firewall events, AI bot rules, and request logs.

Find ChatGPT and Claude crawler requests to WordPress using CDN, host, or web-server access logs.

Use CloudFront request logs to find ChatGPT and Claude crawler paths, user agents, and HTTP outcomes.

Configure nginx access logs to record ChatGPT and Claude crawler paths, user agents, times, and HTTP statuses.

A developer reference for AI crawler user agents, robots.txt tokens, published IP ranges, purposes, and verification methods.

Copyable robots.txt rules for AI crawlers, plus path matching, WAF checks, and a practical verification workflow.

How GPTBot, OAI-SearchBot, and ChatGPT-User differ, which robots.txt rules apply, and how to verify requests in server logs.