llms.txt explained: what it does, who reads it and whether you need one

What is llms.txt?

It is a proposal for a plain Markdown file, placed at the root of a website, that gives language models a short summary of the site and links to its most useful pages. Jeremy Howard of Answer.AI published the idea on 3 September 2024, and the proposal is documented at llmstxt.org.

The reasoning was sensible: web pages are full of navigation, scripts and layout that make them awkward for a model to read inside a limited context window, so a clean, curated summary could help a model find the right material quickly. The proposal was aimed especially at documentation, where a developer or a coding assistant wants the reference pages, not the marketing site.

What the proposal did not do is create an obligation for anyone to read the file. It is a suggestion published by its author, not a standard adopted by search engines, and that distinction is the heart of the debate.

Does Google or any AI search engine use the file?

Not as far as anyone has confirmed. As reported by Search Engine Land, Google's Gary Illyes said Google does not support it, and John Mueller said "none of the AI services have said they're using LLMs.txt", comparing it to the old meta keywords tag that search engines learned to ignore.

We have found no public statement from OpenAI, Perplexity, Microsoft, Anthropic or Google saying their answer engines read the file when deciding what to cite. That may change, and if a major engine documents support we will update this page. Until then, any agency that presents the file as a citation lever is selling a hope, not a documented mechanism.

It is worth comparing that with what the engines do document. OpenAI publishes how OAI-SearchBot follows robots.txt. Perplexity documents PerplexityBot. Anthropic documents Claude-SearchBot. Google documents that AI Overviews and AI Mode rely on Googlebot and its normal index. Those are the controls that have stated effects.

What does Google say you need for AI search instead?

Ordinary SEO. Google's AI optimization guide, updated in July 2026, says optimizing for generative AI search "is still SEO" and that no special files, markup or Markdown are needed.

Google's AI features documentation adds that a page only has to be indexed and eligible to show a snippet to appear in AI Overviews or AI Mode, with no additional requirements and no special optimizations. Google also says there is no special schema.org markup for generative AI search. Put together, Google's position is clear: a curated Markdown summary of your site is not something its AI features need.

How is it different from robots.txt and a sitemap?

Robots.txt and sitemaps are established files that crawlers document and act on; the newer file is a proposal with no confirmed reader among answer engines. The table below sets out the practical differences.

Where is the file actually useful?

Mainly on developer documentation. Several AI companies publish the file for their own developer docs, where it gives coding agents and developers' AI assistants a clean index of reference pages.

If your company ships an API, an SDK or technical docs, publishing one is a reasonable courtesy to developers who paste your docs into an AI tool or use an agent that looks for the file. For a typical marketing site, a services firm or a store, the benefit is small and unproven. It does no harm, it costs little, and that is about the most honest summary.

Should you publish one anyway?

If it takes an hour and you have docs or a clear page hierarchy, go ahead, but budget for it as hygiene, not strategy. Do not pay a retainer for it, and do not expect it to change what ChatGPT or Google AI Overviews say about you. If someone pitches it as the reason you are missing from AI answers, ask them to show the documentation from the engine in question; so far, none exists.

If you do publish one, keep it accurate and maintained. A stale summary that lists retired products or old pricing is worse than none, because it adds one more place where your facts can drift. Link only to pages you want read, make sure those pages are themselves crawlable HTML, and add it to the list of files you review whenever the site structure changes.

Which AI files and settings do engines actually honor?

Robots.txt tokens and, on Cloudflare, managed robots.txt and content signals. These are the places where you decide what AI crawlers may do, and the vendors document how they respond.

The key is to separate search from training. OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so those products can show and cite them; GPTBot, ClaudeBot and CCBot gather training data. Google-Extended is not a crawler at all but a robots.txt control token: Google says it governs Gemini training and Gemini Apps grounding and does not affect inclusion in Google Search, while AI Overviews and AI Mode use Googlebot. Applebot-Extended works the same way for Apple's foundation model training and does not crawl.

Cloudflare's managed robots.txt can add a line such as Content-signal: search=yes, ai-train=no and disallow the main training crawlers for you. That lets a business opt out of training while staying visible in AI search, which is the decision most businesses actually need to make. It is a far more consequential choice than whether to publish a Markdown summary.

What should you fix before thinking about it?

Fix the items that engines have documented, in roughly this order. Each one has a stated or observable effect on whether answer engines can reach and cite you.

• Robots.txt: allow Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot and Applebot; decide separately on training tokens such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot.
• Cloudflare: since 1 July 2025 its default blocks AI crawlers unless they pay, so check the AI crawler and managed robots.txt settings rather than assuming access.
• Server logs: confirm search crawlers receive HTTP 200 responses, not challenges or errors.
• Rendering: serve key content as server-rendered HTML, not only through client-side JavaScript.
• Content: question-style headings with a direct answer underneath, FAQ blocks and visible last-updated dates.
• Entities: consistent company facts in Organization schema, sameAs links, Wikidata and the review or directory sites your buyers use.

How would you know if engines start using it?

Watch for official documentation and your own server logs. If OpenAI, Google, Perplexity, Microsoft or Anthropic add the file to their crawler or webmaster docs, that is the signal to take it seriously.

In the meantime, your access logs will show which user agents request /llms.txt and how often. Requests from a known search crawler would be worth noting, though a fetch alone does not prove the content affects answers. The real test is the same one we use for every AEO change: a fixed prompt set run before and after, compared month on month.