How to Track ChatGPT Crawlers in AWS CloudFront

Choose the log source

CloudFront standard logs record viewer requests over a time period and deliver the files to the destination you configure. Real-time logs are a separate option when a shorter delay is required.

For crawler analysis, keep:

date and time
distribution and host
URI stem
query string only when approved
status
result type
user agent
referrer

AWS documents the user-agent field as the request header that identifies the browser, search engine, or other client.

Enable logs for the distribution

Open the CloudFront distribution and configure standard logging to the supported destination. Confirm that logging covers the production distribution and that the destination retains files long enough for your audit window.

CloudFront encodes some log values. Decode the user-agent field before matching product tokens.

Search for:

OAI-SearchBot, GPTBot, and ChatGPT-User
Claude-SearchBot, ClaudeBot, and Claude-User
PerplexityBot and Perplexity-User

Keep the full raw value even if your report groups requests by provider.

Read the outcome

The HTTP status tells you what CloudFront returned to the viewer:

200: content served
301 or 302: redirected
403: refused
404: path missing
429: rate-limited
5xx: CloudFront or origin failure

Pair the status with the URI and result type. One total request count cannot show whether the crawler reached useful pages or spent its time on redirects and errors.

Add AWS WAF evidence for blocked requests

CloudFront logs tell you the response. AWS WAF logs can provide the matching rule and action when a web ACL handled the request.

Keep the two sources linked by time and request details. This makes a 403 useful without pretending that every 403 came from WAF.

Verify a crawler before creating an allow rule. OpenAI, Anthropic, and Perplexity publish identity guidance, and some publish current network ranges.

Send a controlled feed to SearchSeal

SearchSeal accepts supported access-log data over HTTPS. Send only the fields needed for crawler classification and outcomes.

Don't forward cookies, authorization headers, request bodies, or unapproved query values. Preserve source coverage and sampling information so “no requests observed” doesn't become a false zero.

Frequently asked questions

Which CloudFront fields identify AI crawler traffic?

Use the date, time, host, URI, HTTP status, result type, and user-agent fields together.

Do I need AWS WAF logs?

Use them when you need to know which web ACL rule and action handled a blocked request.

Does a CloudFront 200 prove a ChatGPT citation?

The log establishes successful delivery of that request. Citation happens in a different system and needs separate evidence.