llms.txt
llms.txt is a proposed plain-text file at the root of a website that summarises what the site is and points to its most important pages for AI systems.
The convention was proposed to give language models a concise, curated entry point to a site — closer in spirit to a hand-written index than to robots.txt, which grants or denies access rather than explaining anything.
Adoption by major engines is not universal and should not be assumed. It is cheap to publish and does no harm, but it is a supplement to clear on-page content and structured data, never a substitute.
Why it matters
It costs almost nothing to add and states plainly what your business is and which pages matter. Treat it as a small hedge on an emerging convention rather than a lever with proven effect.
An example
A documentation site publishes /llms.txt containing a two-sentence description of the product followed by links to its twelve most useful pages, each with one line explaining what it answers. It works as a curated table of contents for any system that chooses to read it — and writing it forced the team to decide which twelve pages actually matter.
Common mistakes
- Treating it like robots.txt. It explains rather than permits; it does not grant or deny crawler access.
- Listing every URL on the site. A file pointing at hundreds of pages is a sitemap in another format and communicates nothing extra.
- Expecting it to produce citations on its own. No major engine has committed to reading it.
Questions
- Who proposed llms.txt?
- It was proposed in 2024 as a lightweight convention for giving language models a concise entry point to a website. It remains a proposal rather than an adopted standard.
- Is llms.txt worth adding?
- It costs very little and does no harm, so it is a reasonable small hedge. It should never be prioritised over the things with proven effect: crawler access, server-rendered content and structured data.