Why ChatGPT and Claude Get 403 or 429 Errors

Read the status before changing anything

A 403 and a 429 aren't the same failure. Start with the exact response, not the crawler name.

A 403 means a firewall, bot rule, access rule, authentication check, or application policy refused the request. A 429 means a rate limit decided the client sent too many requests during a defined window.

Keep the request time, hostname, path, user agent, status, and rule match. Without those fields, troubleshooting turns into guessing.

Check each layer in order

1. robots.txt

Fetch the public robots.txt file from the exact hostname. Check the named crawler and requested path. A robots.txt disallow is a policy signal, but it normally does not produce an HTTP 403 by itself.

2. CDN and bot controls

Inspect Cloudflare AI Crawl Control, Vercel Firewall, or the equivalent host controls. Look for a managed AI-bot rule, JavaScript challenge, verified-bot decision, geographic restriction, or rate-limit event.

3. Origin server

Check nginx, Apache, load-balancer, or CloudFront logs. Confirm whether the request reached the origin and which status the origin returned.

4. Application

Look for middleware that expects cookies, a browser session, JavaScript, authentication, or a supported region. Crawlers can fail those checks even when the public page works in your browser.

Verify the identity before allowing it

Do not create a broad allow rule from a user-agent substring alone. A malicious client can call itself GPTBot or ClaudeBot.

Use the provider's current network ranges, your host's verified-bot directory, or supported HTTP message signatures. Preserve the original request evidence so the classification can be corrected.

When verification is unavailable, keep the result labeled as a claimed identity.

Fix the narrowest rule

Change the rule that caused the failure:

allow the verified crawler on public content paths
keep account, admin, and private API paths protected
remove a browser challenge only for the verified identity
tune the rate limit for crawler behavior instead of disabling it globally
preserve logging for allowed and denied requests

OpenAI's crawler troubleshooting guidance specifically calls out robots.txt, CDN and bot mitigation, human-verification checks, 403 responses, and 429 rate limits.

Confirm the result

After the change, wait for a new observed request or run a supported verification flow. Confirm the exact path now receives the intended response.

Once the path returns 200, the access fault is fixed. That status code says nothing about indexing, citation, traffic, or conversion; each needs its own evidence.

Frequently asked questions

Does robots.txt cause a 403 response?

Robots.txt normally expresses policy without generating the 403 itself. The response usually comes from a CDN, firewall, server, authentication check, or application rule.

Should I allowlist GPTBot by user agent?

Not by user agent alone. Combine the identity claim with current provider network evidence or a verified-bot system.

Does fixing a 403 guarantee a ChatGPT citation?

Fixing the 403 removes one access failure. Indexing and citation remain separate downstream events.