How to Track ChatGPT Crawlers on WordPress
Find ChatGPT and Claude crawler requests to WordPress using CDN, host, or web-server access logs.
How to track ChatGPT crawlers on WordPress
WordPress itself is not the reliable observation point for every crawler request. Use the CDN, managed host, or web-server access logs in front of WordPress. Keep the user agent, hostname, path, time, and HTTP status, including requests served from cache.
Audit AI crawler access on WordPress in five steps
To audit AI crawler access on WordPress, check the served robots.txt, plugin settings, CDN or firewall rules and actual request logs, then test the same page path with crawler tools. A robots.txt allowance describes policy; it does not prove that a crawler can retrieve the page.
Updated October 2026.
-
Identify the robots.txt being served. Open /robots.txt on the public hostname and inspect the site's document root through your hosting file manager. WordPress normally serves a virtual robots.txt when no physical file exists; a physical file can take precedence. Confirm which version reaches visitors before editing. WordPress documents its generated output through do_robots().
-
Review SEO and security plugins. Check settings that edit robots.txt or restrict automated requests. WordPress exposes a robots_txt filter that code can use to change generated rules. Record the applicable setting and served rule; a plugin's saved text may differ from a physical or cached file.
-
Inspect CDN and WAF decisions. On Cloudflare or another edge service, review bot policies, custom rules, rate limits and challenge events for the hostname and path. A challenge or denial can stop access even when robots.txt permits it. Check the host firewall too.
-
Filter access logs by AI user agent. Look for GPTBot, OAI-SearchBot and ChatGPT-User in the user-agent field. Keep timestamp, path, source IP and HTTP status. Distinguish successful fetches from redirects, 403 denials and 429 rate limits. Verify the claimed identity using provider network guidance; strings alone can be copied.
-
Test and compare. Use SearchSeal's robots.txt AI crawler checker for the affected path and its AI crawler user agent checker to interpret a logged token. These inspect policy and claimed identity, respectively; neither proves a genuine crawler fetched the page. Recheck the served file and subsequent logs after a fix.
Find the layer that served the request
A WordPress plugin runs inside WordPress. A crawler request can be answered before WordPress runs at all.
Common examples:
- Cloudflare serves a cached page at the edge.
- A managed host serves a full-page cache.
- nginx or Apache blocks the request.
- A security plugin rejects the request after PHP starts.
That's why a plugin-only report can miss real traffic. Start with the outermost layer that records every request to the public hostname.
If the site uses Cloudflare
Open AI Crawl Control and inspect crawler activity for the zone. Check the current Search, Agent, and Training policies, then look at the paths and HTTP outcomes.
Cloudflare can show requests that never reached WordPress. That includes edge cache hits and requests stopped by a firewall rule.
SearchSeal does not connect to Cloudflare or track crawlers or human visits. It saves AI answers with brand mentions and the source links returned with each answer, which you can review separately from these logs.
If the host provides access logs
Ask the hosting panel for web access logs, not only PHP error logs. Each entry should carry a timestamp and hostname, the method and path, the HTTP status, and the complete user agent.
Search for stable identity tokens such as OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot, ClaudeBot, Claude-User, PerplexityBot, and Perplexity-User.
Do not call the identity verified from a string match alone. Check current provider network evidence or your host's verified-bot classification when available.
If you manage nginx or Apache
The standard nginx combined access-log format includes the request, status, referrer, and user agent. Apache combined logs provide a similar set of fields.
Keep query strings out of the crawler feed unless they are needed and approved. They can contain tokens, email addresses, or other values that do not belong in an analytics system.
Keep the raw log under your own retention and access policy.
Why WordPress analytics can disagree
Browser analytics tracks people after JavaScript runs. Security plugins see only requests that reached their code. CDN and server logs see a different part of the request path.
The reports can all be correct and still disagree because each one sees a different part of the request path. Name the layer that produced the number, or the count is meaningless.
Frequently asked questions
Can a WordPress plugin see every AI crawler request?
A plugin can miss requests served or blocked by a CDN, host cache, or web server before WordPress runs.
Where should I check first on a Cloudflare WordPress site?
Check Cloudflare AI Crawl Control and firewall events before WordPress application logs.
Does a crawler user agent prove the request is genuine?
Treat the string as a claimed identity until it is checked against stronger provider or host evidence.
How do I audit AI crawler access on WordPress?
Check whether the public robots.txt comes from a physical file or WordPress virtual output, review SEO and security plugin settings, inspect CDN and WAF rules, filter server or edge logs by AI user agent, then compare the affected path with SearchSeal's robots.txt AI crawler checker and user agent checker. Verify source IPs separately; an allowed rule does not prove successful access.