What is llms.txt?
The open standard that helps AI models navigate and understand your website.
What is llms.txt?
llms.txt is an open standard proposed in September 2024 by Jeremy Howard (founder of Answer.AI and fast.ai). It's a simple Markdown file placed at the root of your website (/llms.txt) that helps Large Language Models understand your site's structure and find key content.
Think of it like robots.txt for traditional search engines, but specifically designed for AI. While robots.txt tells crawlers what they can access, llms.txt tells AI models what they should read to understand your site.
Why does it matter?
- AI tools (ChatGPT, Claude, Perplexity) increasingly consume web content to answer questions
- LLM context windows are too small to process entire websites — llms.txt curates what matters
- Converting HTML (with ads, navigation, scripts) into clean text is difficult — Markdown is native to LLMs
- It gives you control over how AI agents understand your site, products, and documentation
Format & Specification
The file uses Markdown (chosen because LLMs read Markdown natively) with this structure:
# Site Name
> Brief description of your site and what it offers
## Documentation
- [Getting Started](/docs/start): Quick start guide
- [API Reference](/docs/api): Full API documentation
## Optional
- [Blog](/blog): Latest articles and updates
H1 heading (required) — Your site or project name
Blockquote (optional) — Brief summary with key information
H2 sections (optional) — Categories with links in Markdown format
"## Optional" — Content that can be skipped in short contexts
Who uses it?
Major tech companies have already adopted llms.txt:
What about llms-full.txt?
The specification defines a companion file: /llms-full.txt. While llms.txt is a navigation index pointing to content, llms-full.txt contains your entire documentation compiled into a single Markdown file.
AI agents visit llms-full.txt 2x more often than llms.txt, as it provides complete context in one request. Consider providing both for maximum AI visibility.
Best Practices
Place the file at your domain root: yourdomain.com/llms.txt
Keep it updated — sync with documentation releases so AI stays current
Use descriptive notes after each link explaining what the resource covers
Include H2 sections to organize content by category
Mark secondary content with a "## Optional" section
Aim for fast load times (sub-2 second) so AI bots get content before timeout
llms.txt Generator
Fill in the fields below to generate your llms.txt file. Download it and place it at the root of your website.
Sections (H2)
Preview
## Documentation