What llms.txt Is — and What It Isn't

llms.txt is a proposed web standard — created by Answer.AI in late 2024 — that allows website owners to publish a curated, Markdown-formatted list of their most important pages specifically for large language model (LLM) crawlers. The file is placed at the root of a domain (e.g., https://yourdomain.com/llms.txt) and is intended to help AI systems find and prioritize authoritative content more efficiently.

The concept draws on the existing ecosystem of machine-readable files that help automated systems navigate websites:

FilePurposeWho Uses ItOfficial Standard?
robots.txt Tell crawlers which pages to access or avoid All major search engines Yes (RFC 9309)
sitemap.xml List all pages for crawlers to discover All major search engines Yes (sitemaps.org protocol)
llms.txt Curate key pages specifically for LLM crawlers Unconfirmed — no major AI company has confirmed usage No — proposed standard only
Critical Distinction
robots.txt and sitemap.xml are official, widely adopted standards with confirmed support from Google, Bing, and other major search engines. llms.txt is a community proposal that has not been adopted by any major AI company as of April 2026. Treating them as equivalent would be a significant mischaracterization.

The Problem llms.txt Is Trying to Solve

The motivation behind llms.txt is genuine, even if the solution remains unproven. AI crawlers face two structural challenges when processing websites:

  • Context window limitations. LLMs can only process a limited amount of text at once. Most websites contain far more content than fits in a single context window — which means AI systems must make choices about what to read and what to skip. llms.txt proposes to help AI systems make those choices more efficiently by providing a curated index.
  • HTML parsing complexity. Modern websites load significant content via JavaScript, which many AI crawlers cannot execute. Navigation menus, cookie banners, ads, and dynamic content add noise that makes it harder to extract the substantive information. A plain-text Markdown file sidesteps this problem entirely.

There's also a secondary motivation: computational efficiency. Training and running LLMs is expensive. If AI systems could more reliably identify high-quality, relevant content without crawling thousands of low-value pages, the resource savings would be significant. llms.txt is, in part, an attempt to help AI systems work smarter rather than harder.

The Core Idea
llms.txt is essentially a curated table of contents for AI systems — a way of saying "here are the pages on my site that are most worth your attention, and here's what each one covers." The concept is sound. Whether AI systems actually use it is a separate question — and the evidence so far suggests they largely don't.

How llms.txt Files Are Structured

llms.txt files use Markdown formatting — the same lightweight markup language used in GitHub README files, documentation platforms, and many content management systems. Markdown is human-readable, machine-parseable, and doesn't require any special software to create or edit.

Core Markdown Elements Used in llms.txt

  • # Heading — H1 for the site or section name
  • ## Heading — H2 for major content categories
  • > Text — Blockquote for a brief site description
  • - [Text](URL): Description — Linked list items with descriptions

A Complete Example

# llms.txt — example structure # Your Company Name > Brief description of what your company does and who it serves. Important notes: - Key differentiator or important detail about your product - What your product does NOT do (helps AI avoid misrepresentation) - Compliance certifications or security posture (SOC 2, GDPR, etc.) ## Products - [Product Name](https://example.com/product): Core use case and primary benefit - [Pricing](https://example.com/pricing): Plan tiers, starting price, and billing options ## Documentation - [Getting Started](https://example.com/docs/start): Setup guide for new users - [API Reference](https://example.com/api): Complete API documentation with authentication - [Integrations](https://example.com/integrations): Supported third-party tools and connection guides ## Company - [About](https://example.com/about): Company background, mission, and team - [Security](https://example.com/security): Security posture, certifications, and data handling - [Contact](https://example.com/contact): How to reach the team

The specification doesn't mandate a rigid structure — as long as you use valid Markdown, the file is machine-readable. Some teams add more granular subsections (H3s and H4s), tables, or code snippets for technical documentation. Others keep it minimal. Both approaches are valid.

Who Is Using llms.txt in 2026?

Adoption remains niche. According to NerdyData crawl data (April 20, 2026)[1], approximately 4,200 domains had published an llms.txt file as of April 2026 — up from 951 domains in July 2025, but still a tiny fraction of the web's estimated 1.1 billion active domains.

4,200
domains with llms.txt files as of April 2026 (NerdyData, Apr 20, 2026)
4.4×
growth in adoption from July 2025 to April 2026 — but from a very small base
0%
of major AI companies have officially confirmed they use llms.txt files when crawling

The companies that have adopted llms.txt are predominantly developer-focused SaaS brands and documentation-heavy platforms. Here's how some notable adopters have structured their files:

CompanyFile FocusStructure ApproachNotable Characteristic
Hugging Face Developer documentation Multi-level headings (H1–H4), code examples, extensive links Comprehensive knowledge base approach; most detailed of major adopters
Vercel Developer documentation Descriptive metadata lines at top (title:, description:, tags:), then structured sections Adds metadata context before content; step-by-step instructions with code
Zapier Developer documentation Minimal headings; primarily a long list of links with descriptions Lightweight approach; easy to maintain but less contextually rich
Anthropic Company and product information Standard specification format Notable: Anthropic publishes llms.txt but has not confirmed their AI crawler uses it
The Anthropic Paradox
Anthropic — the company behind Claude — has published an llms.txt file on their own website. This is frequently cited as evidence that the format has legitimacy. However, publishing a file and using files published by others are entirely different things. Anthropic has not confirmed that ClaudeBot reads llms.txt files when crawling other websites. The two facts are unrelated.

Does llms.txt Actually Work? What the Evidence Shows

This is the question that matters most — and the honest answer is: the evidence for effectiveness is thin to nonexistent.

What Server Log Analysis Shows

The most direct way to test whether AI crawlers use llms.txt is to analyze server logs and check whether AI bots are actually accessing the file. Multiple independent analyses have been conducted since the format was proposed.

Server log analysis of sites that implemented llms.txt between mid-2025 and April 2026 consistently shows the same pattern: AI crawlers rarely access the llms.txt file. GPTBot (OpenAI's crawler), Google-Extended (Google's AI crawler), PerplexityBot, and ClaudeBot all show near-zero access rates to llms.txt files, even on sites where the file is correctly implemented and accessible.[2]

Traditional search crawlers like Googlebot and Bingbot do occasionally access llms.txt files — but they treat them with no special priority, accessing them at the same rate as any other page on the site.

What Correlation Studies Show

The Authoritas AI Visibility Index (April 24, 2026)[3] analyzed 10,000 domains — 5,000 with llms.txt files and 5,000 without — and found no statistically significant correlation between llms.txt adoption and AI citation volume, citation accuracy, or share of voice in AI-generated answers.

Sites that saw improved AI visibility after implementing llms.txt showed the same improvement trajectory as comparable sites that didn't implement the file — suggesting the gains were attributable to other factors (content quality improvements, schema markup, increased backlinks) rather than the llms.txt file itself.

What Official Statements Say

Google's John Mueller stated on Bluesky in late 2025: "FWIW no AI system currently uses llms.txt." As of April 2026, no major AI company has issued a statement contradicting this position or confirming that their crawlers use llms.txt files.[4]

Evidence Summary
Three independent lines of evidence — server log analysis, correlation studies, and official statements — all point to the same conclusion: llms.txt does not currently influence AI crawler behavior or AI citation outcomes. This may change if major AI companies adopt the standard, but there is no confirmed timeline for that.

Should You Implement llms.txt?

The decision depends on your team's bandwidth, technical capacity, and appetite for experimentation with unproven standards.

Implement if...

You have developer bandwidth and want to experiment. Your site has complex documentation that's hard for crawlers to navigate. You're a developer-focused SaaS brand where early adoption signals technical credibility. You want to be positioned if the standard gains official adoption.

Skip it if...

You have limited developer time and need to prioritize proven AI visibility tactics. You're expecting measurable improvements in AI citations or traffic. You're treating it as a substitute for schema markup, FAQ structure, or comparison content. You need to justify the investment with data.

The opportunity cost matters here. The time spent creating and maintaining an llms.txt file could instead be spent on tactics with confirmed effectiveness: adding FAQ schema to feature pages, implementing SoftwareApplication schema on pricing pages, building HTML-table comparison pages, or creating original research that earns citations. Those investments have a measurable track record. llms.txt does not.

The Right Mental Model
Think of llms.txt the way you'd think of any experimental web standard in its early stages: low cost to implement, unknown upside, no confirmed downside. If you have the bandwidth, it's a reasonable experiment. If you don't, it's a reasonable thing to skip. What it is not is a meaningful AI visibility strategy on its own.

How to Create and Deploy an llms.txt File

If you've decided to experiment with llms.txt, the implementation process is straightforward — but it does require developer involvement to deploy the file correctly.

1
Decide What Content to Feature

Determine which pages or sections of your website should be highlighted for AI crawlers. Keep the file curated and focused — a short list of your most accurate, citation-ready pages is more useful than a second sitemap. For most sites, this means: product or service pages, current pricing, key documentation, about page, and contact page. Avoid including outdated blog posts, thin content, or pages with information that changes frequently.

2
Create the File in Markdown

Open a text editor (Notepad, VS Code, or any plain-text editor) and create a new file named exactly llms.txt. Format it using Markdown. A minimal but complete structure:

# Your Site Name > One-sentence description of what your site or product does. ## Products - [Product Name](https://example.com/product): What it does and who it's for - [Pricing](https://example.com/pricing): Plan tiers and starting price ## Documentation - [Getting Started](https://example.com/docs): Setup and onboarding guide - [API Reference](https://example.com/api): Full API documentation ## Company - [About](https://example.com/about): Company background and mission - [Contact](https://example.com/contact): How to reach the team
3
Upload to the Correct Directory

Place the file in the root directory of your domain so it's accessible at https://yourdomain.com/llms.txt. If the file covers only a subdomain (such as documentation), place it in the corresponding subdirectory. Upload via your hosting control panel (cPanel → File Manager → public_html/) or through your deployment pipeline. After uploading, verify by visiting the URL directly in a browser.

4
Maintain the File Over Time

An outdated llms.txt file pointing to changed or deleted pages is worse than no file at all — it directs AI crawlers to stale or broken content. Review the file quarterly: remove links to outdated pages, update descriptions when product names or features change, and add links to significant new content. Treat it like a living document, not a one-time setup task.

What to Do Instead: Higher-ROI AI Visibility Tactics

If your goal is to improve how AI systems represent your brand in generated answers, the following tactics have a stronger evidence base than llms.txt:

  • FAQ schema on feature and help pages. Structured FAQ markup gives AI systems clean, self-contained answer blocks to extract. This has a confirmed effect on featured snippet selection and is the closest thing to a proven AI extraction signal available today. See: [internal link: How to Add FAQ Schema for AI Visibility].
  • SoftwareApplication schema on product and pricing pages. Machine-readable product metadata reduces ambiguity in how AI systems represent your product category, pricing, and features.
  • HTML-table comparison pages. Comparison content is among the most frequently cited page types in AI-generated SaaS answers. Image-based tables are invisible to AI extraction; HTML tables are not.
  • Consistent product naming across all pages. Entity consistency — using the same name for the same feature across product pages, docs, FAQs, and comparison pages — reduces the entity confusion that causes AI misrepresentation.
  • Original research and data-anchored expert quotes. Content that provides unique, verifiable information earns citations because AI systems can't find it elsewhere. Generic content that summarizes what other sources already say rarely earns citations.
Priority Order
If you're allocating limited technical SEO bandwidth, prioritize in this order: (1) FAQ schema, (2) SoftwareApplication schema, (3) HTML comparison tables, (4) entity consistency audit, (5) original research. llms.txt comes after all of these — if you have bandwidth remaining.

FAQs About llms.txt

Does llms.txt improve AI search visibility?
As of April 2026, there is no confirmed evidence that llms.txt improves AI search visibility. Server log analysis shows AI crawlers rarely access the file, and correlation studies find no statistically significant relationship between llms.txt adoption and AI citation volume or accuracy. The standard may become more relevant if major AI companies officially adopt it, but there is no confirmed timeline for that.
Is llms.txt an official web standard?
No. llms.txt is a proposed standard created by Answer.AI, not an official W3C or IETF standard. It has not been adopted by OpenAI, Google, Anthropic, Perplexity, or any other major AI company as of April 2026. This distinguishes it from robots.txt (RFC 9309) and sitemap.xml (sitemaps.org protocol), which are official standards with confirmed support from major search engines.
Where should I place my llms.txt file?
Place your llms.txt file in the root directory of your domain, accessible at https://yourdomain.com/llms.txt. If the file covers only a subdomain (such as documentation at docs.yourdomain.com), place it in the corresponding subdirectory. After uploading, verify by visiting the URL directly in a browser to confirm it's accessible.
Should I implement llms.txt in 2026?
Only if you have developer bandwidth and want to experiment with an unproven standard. llms.txt is not a confirmed ranking or citation signal for any major AI platform. Prioritize FAQ schema, SoftwareApplication schema, HTML comparison tables, and entity consistency before investing time in llms.txt. If you do implement it, keep the file small, curated, and up to date.
Does Anthropic's Claude use llms.txt files?
Anthropic has published an llms.txt file on their own website, but has not confirmed that ClaudeBot reads llms.txt files when crawling other websites. Publishing a file and using files published by others are entirely different things. No major AI company has confirmed their crawler uses llms.txt files as of April 2026.
Can llms.txt hurt my site's SEO?
There is no evidence that implementing llms.txt negatively affects traditional SEO. The file is a plain-text Markdown document that doesn't interfere with robots.txt, sitemaps, or schema markup. The primary risk is opportunity cost — time spent on llms.txt could be spent on tactics with a stronger evidence base for AI visibility improvement.