Skip to content
Degen

AI SEO

llms.txt Explained

By Mary, Technical SEO Lead 9 min read Last reviewed:

In short

llms.txt is a proposed file at the root of a website that lists its most important pages in markdown, so a language model can find the useful content without parsing navigation and layout. It's a community proposal rather than an official standard, and no major AI provider has confirmed using it.

Short version, because the honest answer matters more than the enthusiasm around it: it's cheap to add, it may do nothing, and the reasons it may do nothing are worth understanding before you spend a morning on it.

What's llms.txt?

A markdown file at /llms.txt describing what your site is and linking to the pages that matter, so a model reading your site doesn't have to work it out from navigation, cookie banners and layout.

The comparison people reach for is robots.txt, and it's misleading. robots.txt restricts. llms.txt recommends — it's a curated table of contents, not a permission file.

Where It Came From

A community proposal from late 2024, picked up quickly by developer-tool and documentation companies. It's not published by a standards body and no search or AI provider is behind it.

What Goes in the File

The proposed format is deliberately simple:

# Company name

> One-line description of what the company does.

## Docs
- [Getting started](https://example.com/docs/start): How to set up in five minutes
- [API reference](https://example.com/docs/api): Full endpoint documentation

## Guides
- [Pricing explained](https://example.com/pricing): How billing works

An H1 with the name, a blockquote summary, then H2 sections of links each with a short description.

llms.txt Versus llms-full.txt

File What it holds Suits
/llms.txt Links plus one-line descriptions Most sites
/llms-full.txt The full text of those pages inline Documentation, where one fetch beats twenty

What Good Ones Have in Common

They're curated, not exhaustive. A sitemap lists everything; this should list the twenty pages that actually answer something. Descriptions say what the page answers rather than what it's called.

Does Anything Actually Read It?

This is the part usually left out of articles about it.

No major AI provider has publicly confirmed using it. Not OpenAI, not Anthropic, not Google. It emerged as a community proposal, was adopted quickly by developer-tool companies, and hasn't been picked up as an official standard by the systems it's aimed at.

What's genuinely true: some documentation platforms generate one automatically, some smaller tools and crawlers do read it, and it costs very little to publish.

How to Check Whether Yours Is Being Read

Filter your access logs for requests to /llms.txt and group by user agent. That's the only honest answer available, and for most sites it's a very short list.

So Should I Add One?

Yes, if you can generate it, with correct expectations.

Reasons To

It takes under an hour, costs nothing to maintain if it's generated from existing content, and may become supported. Writing it also forces a useful question: which twenty pages actually matter?

Reasons Not to Bother Yet

It isn't read by the systems that matter today, and every hour spent on it is an hour not spent on the things that demonstrably work. The mistake is treating it as an AI-visibility strategy. It's a small hygiene item at the end of one.

What to Do Instead

If the goal is being cited by assistants, the ranking isn't close.

  1. Let the crawlers in. GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended — in robots.txt and in your CDN or firewall rules. This blocks everything else when it's wrong, and it's wrong more often than people expect.
  2. Structure pages answer-first. One question per heading, the answer in the opening sentences.
  3. Get mentioned by third parties. The roundups and threads models read are where shortlists come from, and this is the half that decides the outcome.
  4. Then publish llms.txt, because it's twenty minutes.

What This Costs if Somebody Else Does It

Steps one and two are an audit and a rewrite — typically £1,000–£3,000 as a one-off in the market. Step three is ongoing and is where the real budget goes. Ours are on the AI SEO page.

About the author

Mary · Technical SEO Lead

Handles the parts of search nobody sees: crawl access, page structure, log files. Usually finds the problem is a rule somebody set years ago that no one has read since.

FAQ

Frequently asked questions

Is llms.txt an Official Standard?

No. It's a community proposal from late 2024. It has real adoption among developer-tool companies and documentation platforms, but it hasn't been ratified by any standards body or adopted publicly by the major AI providers.

Does ChatGPT Read llms.txt?

There's no public confirmation that it does. Treat any claim otherwise as unverified — including ours if this page ever goes stale.

What's the Difference Between llms.txt and robots.txt?

robots.txt restricts what crawlers may access and is widely honoured. llms.txt recommends what's worth reading and has no enforcement behind it. They serve opposite purposes and aren't alternatives to each other.

Should I Use llms.txt to Block AI Crawlers?

It can't do that — it has no blocking mechanism. Use robots.txt directives for the specific agents, and understand that blocking them also removes you from the answers they generate.

What's llms-full.txt?

A variant that inlines the full text of your key pages rather than linking to them, so a model gets the content in one fetch. It suits documentation sites and is unnecessary for most others.

Keep reading

Be the answer. Not a search result.