Fix a page in six steps
- Run the free AI readiness check. Enter the page URL in the AI readiness check. It runs the same six checks as a SearchSeal page check, with no account, and shows what fails.
- Look up your robots.txt rules. Use the robots.txt AI crawler checker to see how your rules apply to a path for fourteen crawler and control tokens. A rule written years ago can block a bot that AI search uses today.
- Edit robots.txt. Allow the search bots you want to reach the page. OpenAI uses separate crawlers for search and for training, so allowing OAI-SearchBot does not allow GPTBot.
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: /
- Put the main text in the HTML. Open the page source. If the body is an empty container, render the main text on the server so the HTML a crawler receives holds it.
- Add structure and an llms.txt file. Use one H1, at least three headings and 150 or more words of text. Create an llms.txt file with the free llms.txt generator and publish it at your site root.
- Recheck. Run the check again. In SearchSeal a recheck closes the fix automatically when the problem is gone and reopens it if the page fails again.
The free tools
Four free tools cover the work, and none needs an account:
- AI readiness check: page access for ChatGPT, Claude, Perplexity and Gemini, plus schema, llms.txt and structure, with fixes to copy.
- Robots.txt AI crawler checker: a path-specific lookup of your rules for fourteen crawler and control tokens.
- AI crawler user agent checker: the owner, purpose and verification guidance for a user agent string.
- llms.txt generator: builds the file from your sitemap or page URLs, with a preview and a download.
What SearchSeal watches after the fix
SearchSeal checks your site's robots.txt every day. A page check fetches the page HTML, robots.txt and llms.txt and scores six areas: AI crawlers (35 points), schema markup (20), page structure (20), title and description (10), social card (10) and llms.txt (5). Each failing page check becomes a fix in the fix queue. Crawler blocks join one site-wide robots.txt fix, so a blocked page links to that fix and does not open a second one.
Start a check from the Actions page, the REST API or an MCP client. Checks work on your brand's own domain and its subdomains, and each plan has a monthly allowance. Read how they work on the page checks page.
Limits
A page check reads the HTML your server returns and does not run JavaScript. That matches how many crawlers read a page, and it can make a script-heavy page score lower than it looks in a browser.
SearchSeal does not track crawler traffic, so it cannot show whether a bot visited a page after you fixed it. A readable page is not a cited page. After the fix, read the saved answers and their source links to see whether the page appears.
For the crawler tokens and their purposes, see the guides on AI crawler user agents and robots.txt for AI crawlers.
FAQ
Does allowing OAI-SearchBot allow GPTBot?
No. OpenAI uses separate crawlers for search and for training. A robots.txt rule for one does not apply to the other.
Why does a page score low in the check when it looks fine in my browser?
The check reads the server HTML and runs no JavaScript. Text that appears only after scripts load is not counted.
Does SearchSeal show which bots visited my page?
No. SearchSeal does not track crawler traffic. It checks whether robots.txt blocks AI search bots and whether the page is readable.
Are the crawler tools free?
Yes. The readiness check, robots.txt checker, user agent checker and llms.txt generator are free and need no account.