How to Track ChatGPT Crawlers in nginx Logs

The default combined log is a useful start

nginx documents a predefined combined log format containing the client address, request, status, response size, referrer, and user agent.

That's enough to find a claimed crawler and see which path and status it received. A dedicated JSON format is easier to stream and parse.

Use a small JSON log

This example records the fields needed for crawler evidence without cookies, authorization headers, or the query string:

log_format searchseal escape=json
  '{"time":"$time_iso8601",'
   '"host":"$host",'
   '"method":"$request_method",'
   '"path":"$uri",'
   '"status":$status,'
   '"user_agent":"$http_user_agent"}';

access_log /var/log/nginx/searchseal-access.json searchseal;

Test the nginx configuration before reloading it. Confirm the log receives requests for the production server block and that file permissions match the process that reads or ships it.

Match stable identity tokens

Crawler versions can change. Match the stable product token inside the full user-agent value:

OAI-SearchBot, GPTBot, ChatGPT-User
Claude-SearchBot, ClaudeBot, Claude-User
PerplexityBot, Perplexity-User

Keep unknown as a valid classification. Do not force an unfamiliar bot into the closest known name.

Verify the source separately

A user-agent token is not authentication. Compare the connection source with current provider network ranges where published, or use a supported verified-bot system in front of nginx.

If nginx sits behind a CDN or load balancer, define which proxy is trusted and which forwarded address is authoritative. Do not trust an arbitrary X-Forwarded-For value from the public internet.

Read the request as evidence

Group requests by crawler, path, and status:

2xx shows content served
3xx shows redirects
403 shows access refused
404 shows a missing path
429 shows rate limiting
5xx shows a server-side failure

Inspect individual paths before changing a rule. A crawler hitting robots.txt and nothing else tells a different story from the same crawler receiving 200 responses across product pages.

Stream only what SearchSeal needs

SearchSeal accepts supported access-log events over HTTPS and classifies crawler evidence separately from human analytics.

Send the minimum fields required for the job. Query strings, cookies, request bodies, and authorization values don't belong in this feed.

Frequently asked questions

Does the default nginx access log include the user agent?

The predefined combined format includes the request, status, referrer, and user agent.

Should I log the full query string?

Not by default. Query values can contain personal data or secrets and are not required for basic crawler classification.

Can I verify GPTBot from the user agent alone?

The user agent is only a claim. Check current provider network evidence or a verified-bot system separately.