AI SEO
llms.txt Explained
In short
llms.txt is a proposed file at the root of a website that lists its most important pages in markdown, so a language model can find the useful content without parsing navigation and layout. It's a community proposal rather than an official standard, and no major AI provider has confirmed using it.
Short version, because the honest answer matters more than the enthusiasm around it: it's cheap to add, it may do nothing, and the reasons it may do nothing are worth understanding before you spend a morning on it.
What's llms.txt?
A markdown file at /llms.txt describing what your site is and linking to the pages that matter, so a model reading your site doesn't have to work it out from navigation, cookie banners and layout.
The comparison people reach for is robots.txt, and it's misleading. robots.txt restricts. llms.txt recommends — it's a curated table of contents, not a permission file.
Where It Came From
A community proposal from late 2024, picked up quickly by developer-tool and documentation companies. It's not published by a standards body and no search or AI provider is behind it.
What Goes in the File
The proposed format is deliberately simple:
# Company name
> One-line description of what the company does.
## Docs
- [Getting started](https://example.com/docs/start): How to set up in five minutes
- [API reference](https://example.com/docs/api): Full endpoint documentation
## Guides
- [Pricing explained](https://example.com/pricing): How billing works
An H1 with the name, a blockquote summary, then H2 sections of links each with a short description.
llms.txt Versus llms-full.txt
| File | What it holds | Suits |
|---|---|---|
/llms.txt |
Links plus one-line descriptions | Most sites |
/llms-full.txt |
The full text of those pages inline | Documentation, where one fetch beats twenty |
What Good Ones Have in Common
They're curated, not exhaustive. A sitemap lists everything; this should list the twenty pages that actually answer something. Descriptions say what the page answers rather than what it's called.
Does Anything Actually Read It?
This is the part usually left out of articles about it.
No major AI provider has publicly confirmed using it. Not OpenAI, not Anthropic, not Google. It emerged as a community proposal, was adopted quickly by developer-tool companies, and hasn't been picked up as an official standard by the systems it's aimed at.
What's genuinely true: some documentation platforms generate one automatically, some smaller tools and crawlers do read it, and it costs very little to publish.
How to Check Whether Yours Is Being Read
Filter your access logs for requests to /llms.txt and group by user agent. That's the only honest answer available, and for most sites it's a very short list.
So Should I Add One?
Yes, if you can generate it, with correct expectations.
Reasons To
It takes under an hour, costs nothing to maintain if it's generated from existing content, and may become supported. Writing it also forces a useful question: which twenty pages actually matter?
Reasons Not to Bother Yet
It isn't read by the systems that matter today, and every hour spent on it is an hour not spent on the things that demonstrably work. The mistake is treating it as an AI-visibility strategy. It's a small hygiene item at the end of one.
What to Do Instead
If the goal is being cited by assistants, the ranking isn't close.
- Let the crawlers in. GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended — in
robots.txtand in your CDN or firewall rules. This blocks everything else when it's wrong, and it's wrong more often than people expect. - Structure pages answer-first. One question per heading, the answer in the opening sentences.
- Get mentioned by third parties. The roundups and threads models read are where shortlists come from, and this is the half that decides the outcome.
- Then publish llms.txt, because it's twenty minutes.
What This Costs if Somebody Else Does It
Steps one and two are an audit and a rewrite — typically £1,000–£3,000 as a one-off in the market. Step three is ongoing and is where the real budget goes. Ours are on the AI SEO page.
About the author
Mary · Technical SEO Lead
Handles the parts of search nobody sees: crawl access, page structure, log files. Usually finds the problem is a rule somebody set years ago that no one has read since.