How to Implement llms.txt — The Developer's Guide

What Is llms.txt?

On September 3, 2024, Jeremy Howard — co-founder of Answer.AI and fast.ai — published a proposal for a new web standard. Not a new API. Not a new framework. A text file.

The idea is simple: put a Markdown file at /llms.txt on your website that tells AI systems what your site is about, what content matters, and where to find it.

Think of it as robots.txt for the AI era — except instead of telling bots what not to crawl, it tells them what to read.

robots.txt  → "Don't go here"     (bouncer)
llms.txt    → "Start here"        (tour guide)

The spec lives at llmstxt.org and the GitHub repo at AnswerDotAI/llms-txt has 2,200+ stars.

Why It Exists

LLMs have a problem with websites. When a model needs to understand your documentation, product, or API, it has to parse HTML pages full of navigation bars, cookie banners, JavaScript, and sidebar ads. The signal-to-noise ratio is terrible.

Site authors know their content best. A curated Markdown file with the 10-20 most important pages, properly described, gives AI systems a clean entry point — no HTML parsing required.

Who actually reads llms.txt today:

• AI coding assistants (Cursor, Windsurf, Claude Code, GitHub Copilot)
• AI agents and MCP-based tools fetching documentation context
• Developer tools that need structured API references

Who does NOT read llms.txt (yet):

• GPTBot (OpenAI's crawler)
• ClaudeBot (Anthropic's crawler)
• PerplexityBot
• Google-Extended

This matters. The spec was designed for inference time — when an AI is answering a user's question and needs context — not for training-time crawlers that scrape everything regardless. OtterlyAI found that only 0.1% of AI crawler requests touched /llms.txt over 90 days.

Does that mean you shouldn't implement it? No. It means you should understand what it actually does today versus what it might do tomorrow.

The Spec: 5 Minutes to Understand

The entire format is Markdown. Here's the structure:

# Your Company Name

> One-line description of what you do.

Optional context paragraphs with key information
an LLM would need to understand your site.

## Section Name

- [Resource Title](https://example.com/page.md): Brief description
- [Another Resource](https://example.com/other.md): What this covers

## Optional

- [Changelog](https://example.com/changelog.md): Release history
- [Migration Guide](https://example.com/migrate.md): Version upgrades

Required: Only the # heading is required. Everything else is optional but recommended.

The "Optional" section is special — AI systems with limited context windows can skip this section to save tokens. Put your nice-to-have resources here.

Link format: Resources should point to Markdown files (.md) when possible. The spec recommends serving Markdown versions of your HTML pages at the same URL with .md appended.

Real-World Examples

Stripe — The Catalog Pattern

Stripe organizes by product area and includes behavioral instructions:

# Stripe API Documentation

> Complete reference for Stripe's payment processing APIs.

When using Stripe APIs, always default to the latest API version.
Never recommend the legacy Card Element — use Payment Element instead.

## Payments
- [Payment Intents](https://docs.stripe.com/payments/payment-intents.md): Create and confirm payments
- [Checkout Sessions](https://docs.stripe.com/payments/checkout.md): Hosted payment page

## Webhooks
- [Webhook Events](https://docs.stripe.com/webhooks.md): Event types and signatures

Notice the behavioral instructions: "Never recommend the legacy Card Element." This is powerful — you're training the AI on how to represent your product correctly.

Anthropic — The Index + Export Pattern

Anthropic keeps llms.txt slim and links to a comprehensive llms-full.txt:

# Anthropic Documentation

> API documentation for Claude, Anthropic's AI assistant.

## Docs
- [API Reference](https://docs.anthropic.com/api.md): Complete API docs
- [Getting Started](https://docs.anthropic.com/quickstart.md): First API call

For complete documentation, see [llms-full.txt](https://docs.anthropic.com/llms-full.txt)

Next.js — The Versioned Pattern

Next.js includes version metadata and organizes by router type:

# Next.js Documentation
@doc-version: 16.1.6

> React framework for production web applications.

## App Router
- [Routing](https://nextjs.org/docs/app/building-your-application/routing.md): File-based routing
- [Data Fetching](https://nextjs.org/docs/app/building-your-application/data-fetching.md): Server components

llms.txt vs llms-full.txt

When to use both: Your documentation is extensive and wouldn't fit in a single context window. Major platforms (Anthropic, Cloudflare, Zapier) maintain both.

When llms.txt alone works: Your content is compact or already well-structured as Markdown.

Cross-reference them: include a link in llms.txt pointing to llms-full.txt.

Implementation Guide

Static Sites (HTML, Hugo, Jekyll)

Drop the file at your web root:

public/
├── index.html
├── robots.txt
├── llms.txt        ← add this
└── llms-full.txt   ← optional

Next.js

Option 1 — Static file: Place in public/llms.txt.

Option 2 — Dynamic route (auto-updates when docs change):

// app/llms.txt/route.ts
import { NextResponse } from 'next/server';

export async function GET() {
  const content = `# My App

> Description of what your app does.

## Docs
- [API Reference](/docs/api.md): Complete API documentation
- [Getting Started](/docs/quickstart.md): Installation and setup
`;

  return new NextResponse(content, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

PHP (Dynamic from Database or CMS)

<?php
// llms.txt.php — served at /llms.txt via nginx rewrite
header('Content-Type: text/plain; charset=utf-8');

$articles = getPublishedArticles(); // your data source
?>
# <?= SITE_NAME ?>

> <?= SITE_DESCRIPTION ?>

## Services
- Service 1: description
- Service 2: description

## Articles
<?php foreach ($articles as $a): ?>
- [<?= $a['title'] ?>](<?= $a['url'] ?>): <?= $a['description'] ?>
<?php endforeach; ?>

Nginx rewrite to serve it at the clean URL:

location = /llms.txt { rewrite ^ /llms.txt.php last; }

Python (Flask/Django)

# Flask
@app.route('/llms.txt')
def llms_txt():
    content = render_template('llms.txt')
    return Response(content, mimetype='text/plain')

# Django
from django.http import HttpResponse
from django.template.loader import render_to_string

def llms_txt(request):
    content = render_to_string('llms.txt')
    return HttpResponse(content, content_type='text/plain; charset=utf-8')

WordPress

Install one of these plugins:

• Website LLMs.txt — integrates with Yoast/Rank Math
• LLMs.txt Generator

Content Best Practices

Do

• Curate ruthlessly — 10-20 key pages, not your entire sitemap
• Write clear descriptions — "Create and confirm payments" beats "Payment documentation"
• Include behavioral instructions — "Always use v2 of this API" or "Default to TypeScript examples"
• Use definitive language — AI systems prefer "costs $25/mo" over "pricing varies"
• Link to Markdown when possible — cleaner for AI consumption
• Keep it under 10 KB — this is a summary, not a data dump
• Update regularly — stale links and descriptions hurt credibility

Don't

• Dump every page — that's what sitemaps are for
• Use marketing language — "revolutionary AI-powered synergy" helps no one
• Forget the blockquote — the > summary is the most-read part of the file
• Include broken URLs — validate links monthly
• Set and forget — review quarterly at minimum

Validation and Testing

Check your implementation:

• llmstxtchecker.net — format validation
• llmsvalidator.com — structure and link checking

Manual test: Paste your llms.txt content into ChatGPT or Claude and ask: "Based on this llms.txt, what does this company do?" If the AI gives a clear, accurate answer, your file is working.

Monitor access: Check your server logs for requests to /llms.txt:

grep "llms.txt" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn

The Honest Assessment

Google's John Mueller compared llms.txt to the <meta keywords> tag — widely adopted by webmasters but ultimately ignored by search engines. That comparison stings, but it's worth hearing.

The reality today:

• ~950 domains have published llms.txt files (per Semrush analysis)
• No major AI platform has officially confirmed they read them
• No correlation has been found between having llms.txt and getting more AI citations
• The actual consumers are developer tools, not search engines

But here's why you should still implement it:

1. It takes 15 minutes. The cost is nearly zero.
2. Developer tools DO use it. If your audience uses Cursor, Claude Code, or Copilot — and they query your docs — llms.txt helps.
3. It forces you to curate. Deciding which 10-20 pages matter most is a valuable exercise regardless.
4. Standards move slowly. RSS took years to gain traction. HTTPS was "optional" until it wasn't. Early adopters who have clean implementations will benefit when (if) major platforms adopt the spec.

Don't implement llms.txt because it will boost your AI visibility tomorrow. Implement it because it's cheap insurance that makes your content more accessible to the AI tools people are already using.

Quick Start Checklist

1. Create /llms.txt at your web root with the Markdown format above
2. Add an # heading with your company/project name
3. Write a > blockquote summarizing what you do in one sentence
4. List 10-20 key pages under ## section headings with brief descriptions
5. Create /llms-full.txt if your docs are extensive (optional)
6. Validate at llmstxtchecker.net
7. Test by pasting the content into an AI and asking what your company does
8. Monitor access logs monthly
9. Update when you ship new features or deprecate old ones

The entire spec is one page. The implementation is one file. The ROI is unknown but the cost is near zero. That's a bet worth making.

Want to skip the infrastructure? See our managed AI service — we handle deployment, you focus on your business.