What is llms.txt, and is it worth adding to your site?

llms.txt is a text file at the root of a website that gives language models a short summary of the site and links to its important pages. It's a proposal from September 2024, not a standard, and the major search engines haven't said they use it for ranking. For a business website it's ten minutes of work with no guaranteed payoff. For a site with a lot of documentation, it makes more sense.

Where it comes from

The proposal was published on September 3, 2024, on the Answer.AI team's blog. The problem it tries to solve: when an AI assistant wants to learn something from a website, it has to guess which pages matter and strip out HTML full of menus, banners and scripts. The idea is for the site owner to say up front: "these are the useful pages, here they are in brief."

The specification lives at llmstxt.org. It hasn't gone through the IETF or W3C, the bodies that publish web standards, so every AI tool decides on its own whether to read it.

What the file looks like

It goes at /llms.txt and is written in Markdown, plain text with a few formatting marks. It has a set structure:

1. A heading with the name of the site or business. It's the only required element.
2. A one- or two-sentence summary, written as a blockquote.
3. Optionally, a few paragraphs with details on how to use the information.
4. Sections with lists of links, each with a short description.
5. A section called "Optional," with links a model can skip when it's short on space.

The proposal also suggests that important pages have a Markdown version at the same URL with .md added at the end. For a small site, that's usually more work than it's worth.

For an auto repair shop, the file would hold the business name, one sentence on what it does and where, then links to the main services, opening hours, service area, contact page and warranty policy.

How it differs from robots.txt and a sitemap

One thing to keep in mind: robots.txt isn't a security barrier either. The standard states that its rules are not a form of access authorization. llms.txt is even less of one. If you want to hide something, put it behind a password, not in a text file.

Is it worth adding?

It depends on your site.

Documentation or a technical product. This is where it makes the most sense. Developers use AI assistants to read documentation, and a clean summary with good links helps them give correct answers.

A business website or a small shop. You can add it; it does no harm and takes little time. But don't expect extra visits. It matters more that you don't accidentally block bots in robots.txt, that your sitemap is up to date, and that your pages answer questions clearly. We wrote about this in our post on Google's AI summaries.

What it doesn't do. It won't lift you in Google. It doesn't force AI bots to read your site, or to stay away from it. It doesn't replace robots.txt.

If you add one, keep it short and current. An llms.txt that points to deleted pages is worse than none.

If you want to know what actually matters for visibility in AI search, see our SEO and AI page.