LLMs.txt Explained

LLMs.txt: The New Standard for AI-Friendly Documentation

Recently, many developer tools have integrated support for the emerging LLMs.txt web standard. This standard is rapidly gaining traction, but what exactly is it, and why is it significant?

While robots.txt and sitemap.xml cater to search engines, LLMs.txt is designed for large language models (LLMs). It provides website information in a format easily digestible by LLMs.

The meteoric rise of LLMs.txt:

Mintlify's pivotal role: On November 14th, Mintlify incorporated LLMs.txt support into its documentation platform. This single action made thousands of developer tool documentation sets LLM-friendly, benefiting platforms like Anthropic and Cursor. The subsequent announcements on X (formerly Twitter) by Anthropic and others, along with the growing adoption by Mintlify-hosted documentation, significantly boosted the standard's visibility. This momentum spurred the creation of community resources such as directory.llmstxt.cloud (by @ifox) and llmstxt.directory (by @screenfluent), which index LLM-friendly technical documentation. Further contributing to the ecosystem, Mot (creator of dotenvx) developed an open-source generator for dotenvx's documentation, and Eric Ciarla of Firecrawl built a tool that automatically generates the file by scraping websites.

The Origin of LLMs.txt: Jeremy Howard, co-founder of Answer.AI, conceived LLMs.txt to address the context window limitations of AI systems. These systems struggle to comprehend large documentation sites due to their limited processing capacity. Traditional SEO methods, optimized for search crawlers, are ineffective in this context. Directly processing HTML pages overwhelms LLMs with navigational elements, JavaScript, CSS, and other unnecessary information, reducing the space for actual content. LLMs.txt offers a solution by providing LLMs with precisely the information they require in a readily understandable format.

Understanding LLMs.txt Files: LLMs.txt utilizes a specific markdown structure, defining two files:

1. /llms.txt: A concise overview of the documentation navigation, enabling LLMs to quickly grasp the site's structure.
2. /llms-full.txt: A comprehensive file containing the entire documentation content in markdown format.

/llms.txt Example:

<code># Project Name
> Brief project summary

Additional context and important notes

## Core Documentation
- [Quick Start](url): Description of the resource
- [API Reference](url): API documentation details

## Optional
- [Additional Resources](url): Supplementary information</code>

/llms-full.txt Example (excerpt from Cursor's file):

<code># AI Review (Beta)

AI Review is a feature that allows you to review your recent changes in your codebase to catch any potential bugs.

<img src="/static/imghwm/default1.png" data-src="https://img.php.cn/upload/article/000/000/000/174047756651576.png?x-oss-process=image/resize,p_40" class="lazy" alt="AI Review">

... (rest of the documentation) ...</code>

LLMs.txt vs. Existing Standards: Unlike sitemap.xml (which lists indexable pages but doesn't aid content processing) and robots.txt (which controls crawler access but doesn't improve content understanding), LLMs.txt directly addresses AI-specific challenges. It overcomes context window limitations, eliminates unnecessary markup, and presents content in an AI-optimized structure.

Using LLMs.txt with AI Systems: Unlike search engines, current LLMs don't automatically discover LLMs.txt files. You must manually provide the file content to your LLM, either by pasting the URL, copying the content into the prompt, or using a file upload feature. Examples are provided for ChatGPT, Claude, and Cursor.

Generating LLMs.txt Files: Several tools assist in creating LLMs.txt files, including Mintlify (automatic generation for hosted documentation), llmstxt by dotenv (using sitemap.xml), and llmstxt by Firecrawl (website scraping).

The Future of LLMs.txt: LLMs.txt signifies a shift towards AI-centric documentation. Just as SEO became vital for search engine visibility, AI-readable content will be crucial for developer tools and documentation. As adoption grows, we can expect new tools and best practices to emerge, ensuring content accessibility for both humans and AI assistants. Currently, LLMs.txt provides a practical solution for enhancing AI comprehension of web content, especially technical documentation and APIs.

The above is the detailed content of LLMs.txt Explained. For more information, please follow other related articles on the PHP Chinese website!