What Is llms.txt? A Complete Guide to Format, Required Info, and Dynamic Generation [LLMO]

The age of AI search is here. ChatGPT web search, Perplexity, Google AI Overviews -- as more users get information through AI, making sure AI understands your site correctly has become critically important.

That is where llms.txt comes in. If robots.txt tells crawlers "you may / may not crawl this page," llms.txt is a signpost that tells AI "here is what this site covers, and here is where to read the details." There is, however, no evidence that publishing one gets you cited in AI answers, and in this site's own access logs no AI crawler fetched it even once over three and a half days. This guide covers what kind of site it is actually worth adding to.

This article covers everything: the llms.txt format specification, what information to include, the criteria for choosing between static and dynamic generation, and implementation code for major frameworks.

1. What Is llms.txt?

llms.txt is a plain text file (in Markdown format) placed in the root directory of your website, designed to communicate your site's overview, structure, and content catalog to LLMs (Large Language Models).

Basic Information

In One Sentence

robots.txt = "Keep out" / llms.txt = "Here's what our site is about"

While robots.txt is an access control file (allow/deny), llms.txt is a content description file (self-introduction). The two don't conflict -- they're actually meant to be used together.

2. Why You Need It -- How It Differs from robots.txt

You might think, "Can't AI understand my site from sitemaps and meta tags?" That's true to an extent, but llms.txt offers distinct advantages.

Comparing robots.txt / sitemap.xml / llms.txt

Three Reasons You Need llms.txt

Reason 1: LLMs Can't Efficiently Crawl an Entire Site

While Googlebot crawls billions of pages to build a massive index, LLM crawlers don't cover sites as comprehensively. By placing an llms.txt file, you can efficiently tell AI, "Here's the most important content on this site."

Reason 2: It gets you cited in AI answers -- a common claim with no evidence behind it

You will often read that llms.txt makes ChatGPT or Perplexity more likely to cite your site, but there is no data or official vendor statement showing that. Google explicitly says you do not need AI text files to appear in AI Overviews or AI Mode (Google Search Central). It is more accurate to treat llms.txt as a table of contents that helps agents reach the right information quickly than as a way to earn citations; Section 9 checks this against real logs.

For a detailed explanation of LLMO, see "What Is LLMO?"

Reason 3: It Conveys Meta Information That sitemap.xml Can't

sitemap.xml is just a list of URLs. With llms.txt, you can also communicate:

• Your site's theme and areas of expertise
• Summaries and categories for each piece of content
• Your site's update frequency and scale
• Whether you offer multilingual support
• Contact information

3. Format Specification -- What to Write and How

llms.txt is written in Markdown format. Here's the format based on the llmstxt.org specification.

Basic Structure

# Site Name

> A brief description of the site. 1-3 sentences covering the site's theme, audience, and value proposition.

## Section Name 1

- [Page Title](URL): Brief summary of the page

## Section Name 2

- [Page Title](URL): Brief summary of the page
- [Page Title](URL): Brief summary of the page

Format Rules

Practical Example

# My Tech Blog

> My Tech Blog is a software engineering blog covering web development, cloud infrastructure, and AI tools. Updated weekly with practical tutorials and comparisons.

## Main Sections

- [Articles](https://example.com/articles): 50+ in-depth technical articles
- [Tutorials](https://example.com/tutorials): Step-by-step coding tutorials

## Popular Articles

- [React vs Vue in 2026](https://example.com/articles/react-vs-vue): A detailed comparison of React and Vue.js for modern web development, covering performance, ecosystem, and learning curve.
- [Docker for Beginners](https://example.com/articles/docker-beginners): Complete guide to Docker containers for developers who have never used containerization.

## Site Information

- URL: https://example.com
- Sitemap: https://example.com/sitemap.xml
- Languages: en, ja
- Contact: admin@example.com

Important Notes

• Write in English by default. The llms.txt specification and examples are all in English, making it the best practice. For multilingual sites, write in English and specify supported languages with Languages: ja, en, es, .... However, for single-language sites (e.g., a site entirely in French, entirely in Russian, etc.), writing in the site's language is perfectly acceptable. Major LLMs (GPT, Claude, Gemini, etc.) process most languages with high accuracy, and users of such sites will query in the same language, ensuring consistent responses
• Always use UTF-8 encoding. Other encodings like Shift-JIS may cause character corruption
• Use absolute URLs (complete URLs starting with https://)
• Only one H1 per file. Use H2 for section divisions

4. What Information to Include -- Required, Recommended, and Optional

Many people get stuck deciding what to write, so here's a breakdown by priority.

Note that the specification (llmstxt.org) requires only the H1 with the site name; the summary blockquote and the H2 link lists are formally optional. "Required" below means the minimum this guide considers necessary for the file to be worth having.

Required Information (Without These, llms.txt Serves No Purpose)

Recommended Information (Improves AI Understanding)

Optional Information (Nice to Have, but Not Essential)

• Tech Stack: What the site is built with (frameworks, etc.)
• Update Frequency: How often new content is added
• Total Content Count: Number of articles, etc.
• License/Citation Policy: Whether AI is allowed to cite your content
• API Information: If programmatic access is available

Information You Should NOT Include

• Sensitive Information: Admin panel URLs, internal API endpoints, etc.
• Personal Information: Unnecessarily detailed personal data (addresses, phone numbers, etc.)
• Credentials: API keys, passwords, etc.

5. llms.txt vs llms-full.txt

llms-full.txt puts a site's entire content into a single file. It is not part of the llmstxt.org specification; it is a convention that spread because documentation platforms such as Mintlify generate it automatically alongside llms.txt (Mintlify docs). Specification v2 instead recommends providing a Markdown version of each page (page.md or page.html.md) and linking to those from llms.txt, rather than bundling everything into one file.

When You Need llms-full.txt

• Technical Documentation: API references and library docs where providing the full text to AI improves answer accuracy
• Knowledge Bases: FAQ collections and glossaries -- content that's frequently cited in fragments

When You Don't Need llms-full.txt

• Blogs and Media Sites: With many articles, combining everything into one file would be too large. The summaries in llms.txt are sufficient
• E-commerce Sites: Product information is better served through structured data (JSON-LD, etc.)
• Corporate Sites: With few pages, llms.txt alone covers everything

For most blogs and media sites, llms.txt alone is enough. Think of llms-full.txt as something designed for technical documentation.

6. Static File vs Dynamic Generation -- Which Should You Choose?

There are two main approaches to maintaining llms.txt.

Approach 1: Static File

Place a text file directly at public/llms.txt (or an equivalent path).

Pros:

• Simplest implementation (just drop the file in)
• Zero server load
• Works without any framework

Cons:

• Every time you add or update content, you need to manually update the file
• Forgetting to update means stale information is served to AI
• Content counts and categories can drift from reality

Approach 2: Dynamic Generation

The application intercepts requests to /llms.txt, fetches the latest data from the database, and dynamically generates the text.

Pros:

• Always reflects the latest information
• Adding an article automatically updates llms.txt
• Content counts and category names are always accurate

Cons:

• Requires more implementation effort (route definition + controller)
• Each request triggers a database query (mitigated with caching)
• Requires a framework

Decision Criteria

Bottom line: when in doubt, go dynamic. The initial implementation cost is higher, but since you can "set it and forget it," long-term maintenance costs are lower. The worst-case scenario is using a static file and forgetting to update it, leaving AI with outdated information.

7. Implementation -- Code Examples by Framework

Let's look at how to implement dynamic llms.txt generation in major frameworks.

Laravel (PHP)

Route Definition (routes/web.php):

use App\Http\Controllers\LlmsTxtController;

Route::get('/llms.txt', [LlmsTxtController::class, 'index']);

Controller (app/Http/Controllers/LlmsTxtController.php):

class LlmsTxtController extends Controller
{
    public function index()
    {
        $articles = Article::published()
            ->with(['translations' => fn($q) => $q->where('locale', 'en')])
            ->orderBy('published_at')
            ->get();

        $lines = [];
        $lines[] = '# My Site Name';
        $lines[] = '';
        $lines[] = '> Site description here.';
        $lines[] = '';
        $lines[] = '## All Articles (' . $articles->count() . ')';
        $lines[] = '';

        foreach ($articles as $article) {
            $t = $article->translations->first();
            if (!$t) continue;
            $url = 'https://example.com/en/articles/' . $article->slug;
            $lines[] = '- [' . $t->title . '](' . $url . '): ' . $t->meta_description;
        }

        $content = implode("\n", $lines);
        return response($content, 200)
            ->header('Content-Type', 'text/plain; charset=utf-8');
    }
}

The key point is to explicitly set Content-Type: text/plain; charset=utf-8. Without it, the response might be interpreted as HTML.

Django (Python)

# urls.py
from django.urls import path
from . import views

urlpatterns = [
    path('llms.txt', views.llms_txt, name='llms_txt'),
]

# views.py
from django.http import HttpResponse
from .models import Article

def llms_txt(request):
    articles = Article.objects.filter(
        status='published'
    ).order_by('published_at')

    lines = ['# My Site', '', '> Description.', '', '## Articles', '']
    for a in articles:
        lines.append(f'- [{a.title}](https://example.com/articles/{a.slug}): {a.meta_description}')

    content = '\n'.join(lines)
    return HttpResponse(content, content_type='text/plain; charset=utf-8')

Next.js (TypeScript)

// app/llms.txt/route.ts (App Router)
import { NextResponse } from 'next/server'

export async function GET() {
  // Fetch data from DB or CMS
  const posts = await getAllPosts()

  const lines = [
    '# My Site',
    '',
    '> Description.',
    '',
    '## Articles',
    '',
    ...posts.map(p =>
      `- [${p.title}](https://example.com/posts/${p.slug}): ${p.description}`
    ),
  ]

  return new NextResponse(lines.join('\n'), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

WordPress (PHP)

Add to functions.php or as a plugin:

// functions.php
add_action('init', function() {
    add_rewrite_rule('^llms\.txt$', 'index.php?llms_txt=1', 'top');
});

add_filter('query_vars', function($vars) {
    $vars[] = 'llms_txt';
    return $vars;
});

add_action('template_redirect', function() {
    if (!get_query_var('llms_txt')) return;

    header('Content-Type: text/plain; charset=utf-8');
    $posts = get_posts(['numberposts' => -1, 'post_status' => 'publish']);

    echo "# " . get_bloginfo('name') . "\n\n";
    echo "> " . get_bloginfo('description') . "\n\n";
    echo "## Articles (" . count($posts) . ")\n\n";

    foreach ($posts as $post) {
        $url = get_permalink($post);
        $desc = get_the_excerpt($post);
        echo "- [{$post->post_title}]({$url}): {$desc}\n";
    }
    exit;
});

For WordPress, don't forget to re-save your permalink settings (flush rewrite rules) after adding the code.

Static Site Generators (Hugo / Astro, etc.)

Write a script to auto-generate it at build time.

# build-llms-txt.sh (example for Hugo)
#!/bin/bash
echo "# My Site" > public/llms.txt
echo "" >> public/llms.txt
echo "> Site description." >> public/llms.txt
echo "" >> public/llms.txt
echo "## Articles" >> public/llms.txt
echo "" >> public/llms.txt

for file in content/posts/*.md; do
  title=$(grep '^title:' "$file" | sed 's/title: //')
  slug=$(basename "$file" .md)
  desc=$(grep '^description:' "$file" | sed 's/description: //')
  echo "- [${title}](https://example.com/posts/${slug}): ${desc}" >> public/llms.txt
done

Run this before builds in your CI pipeline (GitHub Actions, etc.) and it will auto-update with every deployment.

8. Post-Setup Verification and Testing

After setting up llms.txt, verify the following.

Basic Checklist

Additional Checks for Dynamic Generation

• Check llms.txt after adding a new article: Is the new article reflected?
• Content count is correct: Does "All Articles (27)" match the actual number of published articles?
• Unpublished articles are excluded: Confirm that drafts and scheduled posts don't appear

Validation Tools

There is no official validator for the llms.txt format (as of September 29, 2026). Chrome's Lighthouse has an llms.txt item among its "agentic browsing" audits, but it only checks whether fetching the file returns a server error; if the file does not exist (404), the audit is marked not applicable (Chrome for Developers). Check the contents with the methods below.

• Have ChatGPT or Claude read it: Ask "Read https://yoursite.com/llms.txt and tell me about this site"
• Markdown previewer: Paste the llms.txt content into a Markdown previewer to check that the structure renders correctly
• SEO audit tools: Some SEO audit tools have begun supporting llms.txt presence checks

9. Measured: Do AI Crawlers Actually Fetch llms.txt?

People often say "publish it and AI will read it," but whether anything actually fetches it is something you can check in your own server's access logs. This site (arte.itlibra.com) serves a dynamically generated /llms.txt, so we counted requests by User-Agent from September 26, 07:20 to September 29, 12:54 UTC (about three and a half days, 86,696 requests in total).

/llms.txt was requested only three times in that period: twice from the command line (curl) and once by a site-audit tool. AI crawlers read articles hundreds of times but never opened llms.txt. Many of them did check robots.txt, so it is not that they skip small files; they simply do not look for llms.txt.

This is three and a half days of logs from one site, and other sites or longer periods may look different. Combined with the official statements, though, the picture is clear.

• Google: states that "You don't need to create new machine readable files, AI text files, or markup" to appear in AI Overviews or AI Mode (AI features and your website)
• The specification (v2, August 2026): expects llms.txt to be used at inference time, when an agent needs information while helping a user, rather than for training, and says it is used most heavily by coding agents navigating software documentation (llmstxt.org)
• Chrome Lighthouse: its "agentic browsing" audits include llms.txt, but a page fails only if fetching the file returns a server error. A missing file (404) is marked not applicable and costs nothing (Chrome for Developers)

Deciding Whether to Add One

If you do add one, your own logs are the reliable way to see whether it is used. For access logs in the standard nginx or Apache format (combined), this one line counts the User-Agents that requested /llms.txt, most frequent first.

grep ' /llms.txt ' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

10. Real-World llms.txt Examples

OpenAI, Anthropic and Google (Gemini API) publish llms.txt files for their own developer documentation (listed on the specification page). We found no statistics on adoption rates, so treat the table below as a rough indication of trends.

Types of Sites Leading Adoption

What Makes a Good llms.txt

• Clear, concise overview -- the site's area of expertise is immediately apparent
• All content listed with summaries -- AI can grasp the full picture
• Category structure is explicit -- the information architecture is clear
• Dynamically generated -- always reflects the latest state

What Makes a Bad llms.txt

• Only site name and contact info -- too little information to help AI understand anything meaningful
• Full content text copied in -- llms.txt is a "table of contents," not a "full text." Full text belongs in llms-full.txt
• Left outdated -- claiming "10 articles" when there are actually 50 undermines credibility
• Contains sensitive information -- admin panel URLs or API keys exposed

FAQ