Machine surfaces for AI readers: established and experimental

Websites now expose several machine-readable surfaces. They do not all solve the same problem, and none should be presented as a switch for crawling, indexing, citation, recommendation, or conversion. Separate established protocols from emerging conventions before choosing what to ship.

Established mechanisms

• robots.txt. The Robots Exclusion Protocol defines rules that crawlers are requested to honor. It is not access authorization.
• Sitemaps. A sitemap gives search engines information about URLs and can help discovery. Google says it does not guarantee that URLs will be crawled or indexed.
• Structured data. JSON-LD and other supported formats provide explicit clues about page content. Google says the markup must describe visible content and that eligibility for a search feature is not a guarantee that the feature will appear.
• Markdown media type. RFC 7763 registers text/markdown. Serving a Markdown representation can make a resource easier for a client to parse, but the media type does not require an AI system to fetch or cite it.
• Cite-As. RFC 8574 defines a link relation that marks a preferred URI for permanent citation. It expresses a preference; it does not force a user agent to use that URI.
• IndexNow. The IndexNow documentation describes a URL-change notification. A 200 response means the URL was received, and a 202 response means it was received while key validation is pending. Neither response proves a crawl or index decision.

Emerging conventions

llms.txt is a proposal for a Markdown file at the site root that gives an LLM a curated map of important content. It is not the Robots Exclusion Protocol, does not grant or block access, and does not prove that a provider reads the file.

A Markdown twin at a URL with .md appended is also a convention. It can be useful for documentation, datasets, or other artifacts when the HTML version and the alternate representation are generated from the same source. Document the relationship and serve the right media type. Do not call the twin a citation feed.

Choose the surface for the problem

• Use robots.txt for crawler policy requests and server controls for actual access protection.
• Use sitemaps and stable links to help search discovery.
• Use structured data when it accurately describes visible content.
• Use Markdown and Cite-As for a resource that has a meaningful alternate or preferred reference form.
• Use IndexNow to notify participating search engines after a URL changes.
• Use llms.txt only as a maintained, optional convention with an explicit limitation note.

Generate every representation from one source of truth, test that links and facts match, and monitor the observed result separately. A machine surface can reduce ambiguity for a client. It cannot establish that an assistant crawled, indexed, cited, or recommended the page.