Setting Up llms.txt: Format, Examples and What It Can Do

llms.txt is a proposed plain-text Markdown file, placed at the root of a website, that gives language models a curated summary of the site and links to its most useful content. Jeremy Howard of Answer.AI proposed it on September 3, 2024. It is not an access-control file and not an official standard, and as of mid-2026 Google has said AI systems do not use it.

That combination causes confusion. Many guides present llms.txt as essential for AI visibility. The evidence does not support that. It can still be useful, mainly for developer documentation and for AI tools that fetch it on request, and it takes little effort to create. This guide covers the exact format, a working example, what it does and does not do, and how to decide whether to publish one.

What llms.txt is and is not

	robots.txt	llms.txt
Purpose	Tells crawlers what they may access	Gives language models a curated guide to your content
Format	Plain-text directives	Markdown
Standard status	Formalized as RFC 9309	Community proposal
Controls access	Yes, for crawlers that respect it	No
Used by major AI search products	Yes, for their crawlers	No public confirmation of use

Important: llms.txt does not block or allow anything. To control AI crawlers, you still use robots.txt rules for user agents such as OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot, PerplexityBot and Google-Extended.

What the major platforms have said

Google. Google's AI features documentation says you do not need "new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode. On June 2, 2026, Google's John Mueller described llms.txt as "purely speculative for now," noting that the file has existed for years "yet none of the AI systems use it." He pointed to WebMCP, a Chrome proposal for exposing site actions to AI agents, as a more promising direction, and said the more important concern for most sites is not blocking AI agents.

OpenAI, Anthropic and Perplexity. Their public crawler documentation focuses on robots.txt and user agents. None has publicly stated that its search products use llms.txt for ranking or citation.

Where it is used. Some documentation platforms and developer-tool companies publish llms.txt files so developers can feed documentation into AI coding tools and assistants. In that use, a person or tool fetches the file deliberately.

The llms.txt format

According to the proposal, sections appear in this order:

1. An H1 with the name of the project or site. This is the only required section.
2. A blockquote with a short summary containing key information needed to understand the rest of the file.
3. Zero or more Markdown sections of any type except headings, such as paragraphs or lists, with more detail.
4. Zero or more H2 sections containing file lists: Markdown links to URLs with optional notes.
5. An optional section titled "Optional", whose links can be skipped when a shorter context is needed.

The proposal also suggests offering clean Markdown versions of pages at the same URL with .md appended, for example /docs/quickstart.md.

A working example

# Example Analytics

> Example Analytics is product analytics software for mobile app teams. It tracks user events, funnels and retention for iOS, Android and React Native apps, with a free plan up to 10,000 monthly events.

Key facts:
- Founded in 2019, headquartered in Toronto
- SOC 2 Type II report available on request
- SDKs for iOS, Android, React Native and Flutter

## Product

- [Product overview](https://www.example.com/product.md): What the platform does and who it is for
- [Pricing](https://www.example.com/pricing.md): Plans, limits and what each includes
- [Security](https://www.example.com/security.md): Certifications, data handling and encryption

## Documentation

- [Quickstart](https://www.example.com/docs/quickstart.md): Install the SDK and send your first event
- [API reference](https://www.example.com/docs/api.md): REST API endpoints and authentication

## Optional

- [Changelog](https://www.example.com/changelog.md): Release history
- [Blog](https://www.example.com/blog): Articles and announcements

Illustrative. Use your own approved facts and URLs.

How to set it up

1. Draft the summary from your brand facts. Use the same approved description that appears on your homepage and profiles.
2. Choose the essential pages. Pick 5 to 20 pages that answer the most common questions: product, pricing, security, key docs and comparisons.
3. Create Markdown versions if practical. Many documentation platforms can generate them automatically.
4. Publish at the root. Serve the file at https://yourdomain.com/llms.txt with a text content type.
5. Keep it current. Add it to the checklist for pricing changes, launches and repositioning.
6. Do not change robots.txt because of it. Access control is separate.

Should you publish one?

Worth doing if:

• You have developer documentation that people load into AI coding tools.
• Your site is large and a curated guide helps tools that fetch it on request.
• The cost is low because your platform generates it.

Not worth prioritizing if:

• You have not yet fixed crawler access, brand consistency or content gaps.
• You expect it to improve AI search visibility on its own.

Our recommendation: treat llms.txt as a low-cost, low-certainty addition that comes after the fundamentals. Do not spend significant effort on it, and do not believe claims that it is required.

Common mistakes

Treating it as access control. It blocks nothing and allows nothing.

Letting it go stale. An outdated summary contradicts your site.

Stuffing it with every URL. The point is curation.

Inconsistent facts. The summary should match your approved brand description exactly.

Over-investing. Hours spent perfecting llms.txt are usually better spent on content or crawler access.

A hypothetical example

A hypothetical developer API company publishes an llms.txt file linking to Markdown versions of its quickstart, authentication guide and API reference. Its developer community starts pasting the llms.txt URL into AI coding assistants to get accurate setup help, and support tickets about outdated SDK syntax decline. The file did not change the company's visibility in AI search results, but it made documentation easier for AI tools to consume when users requested it.

How Bob Builds AI helps

Bob Builds AI's GitHub integration can open pull requests for llms.txt, schema and technical GEO fixes, and its Linear integration can create engineering issues for them. Agent Analytics helps you see how AI crawlers access your site, which matters more for visibility than any single file.

---

FAQ

What is llms.txt?

llms.txt is a proposed Markdown file placed at a website's root that gives language models a curated summary of the site and links to key content. It was proposed by Jeremy Howard in September 2024. It is not an official standard and does not control crawler access.

Does Google use llms.txt?

No. Google's documentation says no special AI text files are needed for AI Overviews or AI Mode, and in June 2026 Google's John Mueller described llms.txt as "purely speculative for now," saying none of the AI systems use it.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which URLs they may access and is formalized as an internet standard. llms.txt is a content guide in Markdown and has no access-control function. Use robots.txt to manage AI crawlers.

Does llms.txt improve visibility in ChatGPT?

There is no public evidence that it does. OpenAI's crawler documentation focuses on robots.txt and OAI-SearchBot access. Allowing OAI-SearchBot and publishing clear, accurate content are the documented requirements for appearing in ChatGPT search.

What should an llms.txt file include?

An H1 with the site name, a blockquote summary, optional details, H2 sections listing key URLs with short descriptions, and an optional "Optional" section for secondary links. Many sites also provide Markdown versions of the linked pages.

Who benefits most from llms.txt?

Companies with developer documentation benefit most, because developers and AI coding tools can fetch the file to get accurate, current docs. For most marketing sites, it is a low-priority addition.

Can llms.txt hurt my site?

Not directly. The main risk is maintaining an outdated summary that contradicts your site, or spending time on it instead of higher-impact work like crawler access and content quality.

---

Conclusion

llms.txt is easy to create, but it is not a lever for AI search visibility today. Google says AI systems do not use it, and no major AI search provider has confirmed that they do. Its practical value lies in curated documentation that people and tools fetch on purpose.

If you publish one, generate it from your approved brand facts, keep it short and current, and leave access control to robots.txt. Then put most of your effort into crawler access, content clarity and third-party authority, where the evidence is stronger. Bob Builds AI can help you prioritize those fixes and route them to your team.