Fix pages AI crawlers cannot read

An AI tool can only use a page it can fetch and read. Three problems stop that most often: a robots.txt rule that blocks an AI search bot, a page that arrives as an empty shell until JavaScript runs, and a page with almost no structure. The free crawler tools find each one, and SearchSeal page checks keep watching after you fix it.

Check your site
Evidence trail
  • Free robots.txt and user agent checkers
  • Six page checks on any URL of your site
  • Automatic recheck after a fix

Fix a page in six steps

  1. Run the free AI readiness check. Enter the page URL in the AI readiness check. It runs the same six checks as a SearchSeal page check, with no account, and shows what fails.
  2. Look up your robots.txt rules. Use the robots.txt AI crawler checker to see how your rules apply to a path for fourteen crawler and control tokens. A rule written years ago can block a bot that AI search uses today.
  3. Edit robots.txt. Allow the search bots you want to reach the page. OpenAI uses separate crawlers for search and for training, so allowing OAI-SearchBot does not allow GPTBot.
    User-agent: OAI-SearchBot
    Allow: /
    
    User-agent: PerplexityBot
    Allow: /
  4. Put the main text in the HTML. Open the page source. If the body is an empty container, render the main text on the server so the HTML a crawler receives holds it.
  5. Add structure and an llms.txt file. Use one H1, at least three headings and 150 or more words of text. Create an llms.txt file with the free llms.txt generator and publish it at your site root.
  6. Recheck. Run the check again. In SearchSeal a recheck closes the fix automatically when the problem is gone and reopens it if the page fails again.

The free tools

Four free tools cover the work, and none needs an account:

What SearchSeal watches after the fix

SearchSeal checks your site's robots.txt every day. A page check fetches the page HTML, robots.txt and llms.txt and scores six areas: AI crawlers (35 points), schema markup (20), page structure (20), title and description (10), social card (10) and llms.txt (5). Each failing page check becomes a fix in the fix queue. Crawler blocks join one site-wide robots.txt fix, so a blocked page links to that fix and does not open a second one.

Start a check from the Actions page, the REST API or an MCP client. Checks work on your brand's own domain and its subdomains, and each plan has a monthly allowance. Read how they work on the page checks page.

Limits

A page check reads the HTML your server returns and does not run JavaScript. That matches how many crawlers read a page, and it can make a script-heavy page score lower than it looks in a browser.

SearchSeal does not track crawler traffic, so it cannot show whether a bot visited a page after you fixed it. A readable page is not a cited page. After the fix, read the saved answers and their source links to see whether the page appears.

For the crawler tokens and their purposes, see the guides on AI crawler user agents and robots.txt for AI crawlers.

FAQ

Does allowing OAI-SearchBot allow GPTBot?

No. OpenAI uses separate crawlers for search and for training. A robots.txt rule for one does not apply to the other.

Why does a page score low in the check when it looks fine in my browser?

The check reads the server HTML and runs no JavaScript. Text that appears only after scripts load is not counted.

Does SearchSeal show which bots visited my page?

No. SearchSeal does not track crawler traffic. It checks whether robots.txt blocks AI search bots and whether the page is readable.

Are the crawler tools free?

Yes. The readiness check, robots.txt checker, user agent checker and llms.txt generator are free and need no account.

Find your next AI search win

Check your site