How to Track ChatGPT Crawlers on Vercel

Start with the AI Bots Ruleset

Vercel provides an AI Bots Managed Ruleset on all plans. It can identify known AI crawler traffic and apply one of two useful actions:

Log records matched traffic without blocking it.
Deny blocks matched traffic.

Open the project, then go to Firewall, Rules, and Bot Management. Check the action on the AI Bots Ruleset before reading the results. If the rule is set to Deny, a missing origin request may be expected.

Vercel maintains the list behind the managed rule. You don't need to keep adding every new crawler token to a custom expression.

Inspect what Vercel observed

Use Firewall observability to inspect matching events and mitigations. For requests that reached a function, Runtime Logs expose request details including the user agent and firewall decision.

For each event, retain the request time and production hostname, the requested path and user agent, the HTTP status, and the firewall action.

Check OpenAI identities separately. OAI-SearchBot, GPTBot, and ChatGPT-User do different jobs. The same applies to Anthropic's Claude-SearchBot, ClaudeBot, and Claude-User.

Use Log Drains when you need durable evidence

Vercel Log Drains are available on Pro and Enterprise plans. Their proxy log schema can include method, host, path, user agent, status, client IP, and referrer.

Sampling matters. A drain that samples ten percent of traffic can't prove that no crawler visited a page. Preserve the sampling configuration with the report or collect all requests for the paths you need to audit.

Do not retain more personal data than the crawler workflow requires. SearchSeal discards raw IP addresses, full user agents, and unapproved query values at collection after using the request for classification.

Where the Vercel view ends

Vercel's firewall decision covers Vercel. Add Cloudflare or a second host, and the dashboard cannot show what happened outside the project. SearchSeal's crawler module is built for that cross-host evidence trail.

Read an empty result carefully

No matching event can mean:

the crawler did not request the site
the selected time range was wrong
the request hit a hostname outside the project
the relevant source was not retained
sampling dropped the request
a different security layer acted first

Call the result unknown until source coverage is known.

Frequently asked questions

Does Vercel detect AI crawlers?

Vercel provides an AI Bots Managed Ruleset that can log or deny identified AI bot traffic.

Should I use Log or Deny mode?

Use Log when you want to observe matched requests without blocking them. Use Deny only when blocking matches your access policy.

Are Vercel Log Drains available on the free plan?

Vercel documents Log Drains as a Pro and Enterprise feature.

Does Vercel Log mode block AI crawlers?

Log mode records matches without blocking them. Deny mode is the action that blocks matched traffic.