Technical · 6 min read

What is llms.txt and does your website need one?

A plain-English guide to llms.txt: what it is, what it can and cannot do, and how to write one in under an hour.

By the AEO GEO Marketing team · Updated

Key takeaways
  • llms.txt is a proposed Markdown file that summarises a website for large language models.
  • It is optional and not used by Google Search; support among AI tools varies.
  • It costs little to create and can help tools that read it understand your site quickly.
  • A good file has a short summary, key facts and links to your most important pages.
  • It complements, not replaces, robots.txt, sitemaps and good content.

What is llms.txt?

llms.txt is a proposed standard for a plain-text file, written in Markdown, placed at the root of a website (for example, example.com/llms.txt). It gives large language models and AI tools a concise overview of the site: what the organisation does, the most important facts and links to the pages that matter most, with short descriptions.

The idea is simple. Websites are built for human browsers, with navigation, scripts and layouts that make it harder for AI tools to find the core information quickly. llms.txt offers a clean, curated summary designed for machines, similar in spirit to how robots.txt and sitemaps help crawlers.

What llms.txt is not

It is important to be realistic:

  • It is not a ranking factor in Google Search. Google has said it does not use llms.txt.
  • It is not a permission file. Crawl permissions belong in robots.txt.
  • It is not universally supported. Some AI tools, developer assistants and agents read it; many crawlers do not rely on it.
  • It does not replace good content. Engines still cite the pages themselves.

Think of it as a low-cost, low-risk addition that may help some tools understand your site, especially agents and assistants that fetch your site on a user’s behalf.

Why some businesses add it anyway

There are practical reasons to create one:

  • It takes little time and has no downside when written accurately.
  • AI agents that browse on users’ behalf can find key facts quickly.
  • It forces you to define the facts you want AI systems to repeat: what you do, prices, locations and contact details.
  • It provides a useful index for developers and partners.
  • Support may grow as AI tools mature.

A related file, llms-full.txt, contains the full text of key pages in one document for tools that want everything at once.

What to include

A good llms.txt follows a simple structure:

  1. A heading with your organisation’s name.
  2. A short summary in a blockquote: one or two sentences describing what you do and for whom.
  3. Key facts: services, prices, locations, contact methods, response times, policies.
  4. Sections of links to your most important pages, each with a one-line description: core pages, services, guides, locations.
  5. An optional section for less essential resources.

Keep it accurate, factual and free of marketing hype. Everything in it should match your website.

An example structure

A service business might write:

  • A title line: “# Example Agency”.
  • A summary line: “> Example Agency helps mid-sized law firms win clients through search, with retainers from $2,000 per month.”
  • A key facts list: services and starting prices, cities served, contact options, contract terms.
  • A “Core pages” section linking the home, services, pricing, about and contact pages.
  • A “Services” section with one line per service page.
  • A “Guides” section linking your best educational content.
  • An “Optional” section linking a full-text version and the sitemap.

Our own site publishes both llms.txt and llms-full.txt generated from the same data as the website, so they never drift out of date.

How to add llms.txt to your site

  1. Write the file in Markdown following the structure above.
  2. Save it as llms.txt and upload it to the root of your domain.
  3. Make sure it is served as plain text and publicly accessible.
  4. Optionally reference it from robots.txt as a comment and from your HTML with a link element.
  5. Regenerate or update it whenever key facts or pages change.

On WordPress, plugins and some SEO tools can generate it. On static or custom sites, generate it at build time from the same data as your pages, which avoids inconsistencies.

Common mistakes

  • Listing every page on a large site instead of the ones that matter.
  • Including claims or prices that differ from the website.
  • Writing promotional copy instead of factual descriptions.
  • Forgetting to update it after changes.
  • Treating it as a substitute for robots.txt or a sitemap.

What matters more than llms.txt

If you have limited time, prioritise the things that definitely affect AI visibility:

  • Allow the AI search crawlers you want in robots.txt and your firewall.
  • Make key content crawlable in HTML.
  • Write answer-first pages with specific facts.
  • Add accurate structured data.
  • Build reviews and independent mentions.

Then add llms.txt as a finishing touch.

How llms.txt fits with other files

  • robots.txt tells crawlers what they may access.
  • sitemap.xml lists the URLs you want indexed.
  • llms.txt summarises the site and points to the most important content for AI tools.
  • Structured data describes each page’s content in a standard vocabulary.

Together they make your site easier for every kind of machine to understand.

How AI agents might use llms.txt

AI agents are assistants that act on a user’s behalf: researching suppliers, comparing prices or completing tasks on websites. When an agent visits a site, it needs to find key information quickly and reliably. A well-written llms.txt gives it a map: what the business does, where the pricing page is, how to get in touch and which pages contain detailed answers.

As agents become more common, small things like this may make a difference to whether an agent completes a task on your site or moves on to a competitor whose information is easier to find.

llms.txt for different types of sites

Service businesses

Focus on services, prices, locations, contact methods and proof such as case studies and reviews.

E-commerce

Link to main categories, buying guides, shipping, returns and contact pages rather than listing every product.

SaaS and developer tools

Link to documentation, API references, pricing, integrations and changelogs. Developer tools are where llms.txt is most widely used today.

Publishers

Link to topic hubs, editorial standards and the most important evergreen content.

Keeping llms.txt accurate

An out-of-date llms.txt can do more harm than good, because it may repeat old prices or discontinued services. The safest approach is to generate it automatically from the same source of truth as your website. If that is not possible, add it to the checklist for any change to pricing, services or contact details, and review it quarterly.

Write factual statements that can be checked against your pages. Avoid superlatives such as “the best” or “number one”, which add nothing for AI tools and may reduce trust.

Should you also publish llms-full.txt?

llms-full.txt contains the full text of your key pages in one Markdown document. It is useful for tools that want to load everything at once, such as coding assistants reading documentation. For most business sites, a curated llms.txt is enough. If you publish llms-full.txt, keep it focused on your most important content, exclude templated or low-value pages and regenerate it whenever content changes.

Writing descriptions that help AI tools

Each link in llms.txt should have a one-line description that tells an AI tool exactly what it will find on the page. Compare “Pricing” with “Pricing: Growth ($2,000/mo), Scale ($3,500/mo) and Enterprise plans, what each includes, contract terms and payment methods.” The second lets a tool decide whether to open the page and often answers simple questions on its own. Use plain language, include the key facts and keep each description to a single sentence. Descriptions written this way also make a useful checklist of whether each page really contains what it promises.

Where llms.txt came from

The llms.txt proposal was published in 2024 by Jeremy Howard of Answer.AI as a lightweight way for websites to offer LLM-friendly content. It was designed to work at inference time, when an assistant or agent is answering a specific question, rather than for training. Adoption grew first among documentation sites and developer tools, where coding assistants benefit from a clean index of docs, and later spread to marketing sites. Its future depends on how widely AI products choose to read it, which is why it is best treated as a helpful extra rather than a core requirement.

How to tell whether AI tools read your llms.txt

Check your server logs or analytics for requests to /llms.txt and note the user agents. This shows which tools fetch it and how often. Combined with your normal AI visibility tracking, it gives a realistic picture of whether the file is being used, so you can decide how much effort to invest in maintaining it.

Key terms in this guide

  • LLM (Large Language Model): An AI model trained on large amounts of text to understand and generate language. LLMs power ChatGPT, Gemini, Claude, Copilot and many AI search features.
  • Citation: A link or reference to a source shown inside an AI-generated answer. Citations are the main unit of success in GEO.
  • llms.txt: A proposed plain-text file at a website’s root that gives AI systems a concise, structured overview of the site and links to its most important content.
  • robots.txt: A file that tells crawlers which parts of a website they may access. It can allow or block specific AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.

How AEO GEO Marketing can help

If you would rather have specialists handle this, these services cover the work described in this guide:

AI SEO & Generative Engine Optimization

Get cited in Google AI Overviews, ChatGPT & Perplexity. The new search visibility standard. It includes geo / aeo full audit, ai overview citation strategy, entity authority building and more, from $800/mo.

Web Design & Development

Performance-first websites built for conversion, speed and AI search visibility. It includes ux & visual design, core web vitals engineering, schema-ready architecture and more, from $1,500/mo.

AI Visibility Audit

A full report on how AI engines see your brand, with a prioritised 90-day action plan. It includes prompt testing, competitor benchmark, technical ai readiness and more, from $750 one-off.

Not sure where to start? The free 48-hour AI snapshot shows where AI engines cite you today.

Frequently asked questions

Is llms.txt an official standard?

It is a community proposal that has gained adoption among developers and some AI tools. It is not an official web standard.

Does ChatGPT read llms.txt?

OpenAI has not stated that its search uses llms.txt for ranking or citations. Agents and tools may fetch it when visiting a site.

Can llms.txt hurt my SEO?

No, provided it is accurate. Search engines treat it as an ordinary text file.

How long should it be?

Usually a few hundred to a few thousand words. Large sites should curate rather than list everything.

Continue with these guides:

Or see how we put this into practice:

Want this done for you?

Get a free 48-hour GEO audit that shows exactly where AI engines cite you — and where they cite your competitors.