The honest answer to whether llms.txt does anything yet is: almost nothing, measurably, and the people telling you it is essential for AI visibility are selling something. That is an uncomfortable position to hold while also recommending you publish one, so this post shows the numbers and then explains the recommendation.

The file itself is a sensible idea. It is a markdown index at the root of your site telling a language model what you publish and where, in the way robots.txt tells a crawler what it may fetch. The idea is fine. The adoption by the companies it was designed for is the problem.

What the server logs actually show: an Ahrefs analysis of 137,000 domains found that 97% of llms.txt files received zero requests in May 2026. Not few requests. Zero. Meanwhile adoption grew 8.8 times in a year, which means the file is being published far faster than it is being read.

The Numbers, and Where They Come From

Two datasets are worth knowing, because most articles on this subject cite neither.

Originality.ai tracked more than three million websites between June 2025 and May 2026. Instances of llms.txt went from 4,088 to 36,120, an 8.8x increase, with roughly 38,980 sites publishing one of the related formats by May 2026. Real growth, from a very small base.

Ahrefs then analysed server logs across 137,000 domains and found 97% of those files received zero requests in May 2026. Of the requests that did arrive, AI retrieval bots accounted for 1.1%. GPTBot made 4.51% of requests to these files, ClaudeBot 0.80%, DeepseekBot 0.02%.

Those two studies together describe a standard being adopted enthusiastically by publishers and ignored comprehensively by consumers.

What the AI Companies Actually Say

This is the part that settles the argument.

Google does not support it. Google said so publicly in July 2025 and said there were no plans to. OpenAI does not mention llms.txt in its crawler documentation and points site owners at robots.txt. Anthropic publishes its own llms.txt and has engaged with the proposal, but has not stated that Claude’s retrieval parses third-party files.

Perplexity is the notable exception, saying it retrieves the file to help prioritise pages.

There is a delicious irony buried in the data: Google, OpenAI and Anthropic all publish llms.txt files for their own documentation while advising site owners not to rely on it for search visibility. It is useful for their docs to be machine-readable. That is not the same as their crawlers reading yours.

The genuine consumers today are coding agents rather than search engines. Cursor, Claude Code, Continue and Cline read llms.txt when a user points them at a domain. If your product has developer documentation, that is a real audience with real intent, and it is probably the strongest argument for having one.

So Why Publish One At All

Four reasons, none of them “it will improve your rankings”.

It costs almost nothing. A generated index of your own content is an afternoon, and less if your site already knows its own structure.

The consumers that exist are high-intent. A developer pointing Cursor at your docs is closer to using your product than most search traffic. Perplexity prioritising your pages is small but real.

Standards adoption is a ratchet. If retrieval systems do start consuming it, sites with a correct, current file are ready and sites without one are not. The cost of being early is one afternoon; the cost of being late is however long it takes you to notice.

Writing it forces an audit. Producing an accurate index means enumerating everything you publish, which is how you find the pages nobody links to and the sections you forgot existed.

That last one is not theoretical. Auditing our own file this month turned up a problem worth more than the file itself.

A Stale llms.txt Is Worse Than No llms.txt

Our llms.txt listed 29 posts. The site had 166. Worse, every URL in it omitted the language prefix, so mecanik.dev/posts/<slug>/ returned 404 where the live page was at mecanik.dev/en/posts/<slug>/. Of 243 site URLs in the file, seven resolved.

Anything that did read it was handed a map of a site that did not exist. That is strictly worse than publishing nothing, because nothing at least fails honestly.

The lesson generalises past this one file. An index is a promise about your site, and an unmaintained promise becomes a lie quietly. If you publish one, put it in the same release process as your sitemap, and test that the URLs resolve rather than assuming the generator got them right. A slug existing in your content is not the same as a URL resolving on your server.

How to Publish One Properly

Follow the llmstxt.org specification rather than inventing a format. An H1 with the site name is the only required element; a blockquote summary and H2 sections of annotated links are the convention. There is an Optional section by convention for links an agent can skip when context is short.

Beyond the spec, three practical rules. Generate it from your content rather than maintaining it by hand, because a hand-maintained index is a stale index within two months. Test every URL with a real request. And if your site is multilingual, state the language pattern once rather than listing every locale, so a reader can construct any URL from a rule.

Our generative engine optimisation guide covers the parts of AI visibility that do measurably work, and how AI search engines read structured data covers schema, which unlike llms.txt is consumed by systems that publicly document consuming it.

The Recommendation

Publish one. Keep it accurate. Expect nothing from it this year, and do not let anyone bill you for it as an AI visibility service.

If you want the thing that actually moves AI citations, it is not a file at the root of your domain. It is having content that answers a question completely enough to be quoted, and enough authority that a retrieval system trusts the answer. Mecanik covers both in a technical SEO audit , including whether your machine-readable indexes are telling the truth about your site.

Related reading: Why Your Content Ranks But Never Gets Cited , Blocking or Allowing AI Crawlers: The Business Decision , Google AI Mode: What It Means for Your Website Traffic and Law Firm SEO Agency: Compliance, Content and Cost .

Frequently Asked Questions

Does llms.txt actually work in 2026? Barely. An Ahrefs analysis of 137,000 domains found 97% of llms.txt files received zero requests in May 2026, and AI retrieval bots made up 1.1% of the requests that did arrive. Adoption grew 8.8 times in a year, so the file is being published far faster than anything reads it.

Do Google, OpenAI and Anthropic support llms.txt? No. Google said publicly in July 2025 that it does not support llms.txt and has no plans to. OpenAI does not mention it in crawler documentation and points site owners at robots.txt instead. Anthropic publishes its own file but has not said Claude’s retrieval parses third-party ones. Perplexity is the exception and says it retrieves the file to prioritise pages.

Is llms.txt the same as robots.txt? No. robots.txt tells crawlers what they may fetch and is universally honoured by well-behaved bots. llms.txt is a content index telling a language model what you publish and where, and it is honoured by very little. If you want to control AI crawler access, robots.txt and your CDN’s crawler controls are the mechanisms that work.

Who actually reads llms.txt today? Mostly coding agents rather than search engines. Cursor, Claude Code, Continue and Cline read the file when a user points them at a domain, and Perplexity says it uses it to prioritise pages. If you publish developer documentation, that is a small but genuinely high-intent audience.

Should I publish an llms.txt file? Yes, but for the right reasons. It costs an afternoon, the consumers that do exist are high-intent, and you are ready if adoption improves. Expect no ranking benefit. Most importantly, keep it accurate: a stale file that lists URLs which no longer resolve is worse than publishing nothing, because it hands any reader a map of a site that does not exist.