Should you add llms.txt to your website?
Should you update robots.txt for ChatGPT, Claude and Perplexity?
And if you have one, do you still need the other?
These questions are increasingly being bundled together as brands prepare their websites for AI discovery.
But llms.txt and robots.txt do fundamentally different jobs.
One tells crawlers where they are permitted to go.
The other can help an agent understand where useful content lives.
Confusing the two can create a false sense of AI readiness. A beautifully organised llms.txt file does not help if the crawler is blocked from the pages it points to. And allowing a crawler through robots.txt does not mean the page it reaches will contain useful, machine-readable information.
The more useful question is not:
“Which file should we use?”
It is:
“What problem is each file actually solving?”
robots.txt answers: can this crawler access the page?
robots.txt is the established mechanism websites use to communicate crawling preferences to automated clients.
The Robots Exclusion Protocol was formalised as an Internet standard in RFC 9309. It defines how crawlers identify themselves through a user agent and how website owners can use rules such as Allow and Disallow to indicate which URL paths may be crawled.
A basic file might look like this:
User-agent: ExampleBot
Disallow: /private/
Allow: /public/The important word here is access.
robots.txt does not explain the meaning of your website.
It does not summarise your products.
It does not make a JavaScript page readable.
And it is not a security control. RFC 9309 explicitly notes that robots rules are not access authorisation. Sensitive content still requires proper authentication or other security controls.
What robots.txt does is give participating crawlers a policy for which URLs they should and should not fetch.
That role remains highly relevant to AI systems.
The major AI platforms still document robots.txt
Look at how the major AI platforms currently tell website owners to manage crawler access.
OpenAI
OpenAI distinguishes between crawlers used for different purposes.
For visibility in ChatGPT search, its current publisher guidance tells website owners to ensure they are not blocking OAI-SearchBot, including updating robots.txt where necessary.
GPTBot is treated separately because it relates to potential model training rather than ChatGPT search discovery.
OpenAI’s publisher and crawler guidance
Anthropic
Anthropic similarly documents separate agents including ClaudeBot, Claude-SearchBot and Claude-User.
Its guidance says its bots honour crawling directives expressed through robots.txt, allowing website owners to make different choices depending on whether the use case is model development, search or user-directed retrieval.
Perplexity
Perplexity documents PerplexityBot for search discovery and recommends allowing it through robots.txt if a website wants to appear in Perplexity search results.
It separately documents Perplexity-User for user-requested retrieval. Perplexity also notes that organisations using a WAF may need to allow its published IP ranges in addition to configuring crawler rules.
Perplexity’s crawler documentation
So when the question is:
“How do I control whether a documented AI crawler can crawl my site?”
robots.txt remains the relevant mechanism.
llms.txt answers a different question
llms.txt was proposed in 2024 as a way to give language models and agents a concise, Markdown-based guide to a website.
Rather than defining permissions, it provides context and links to useful resources.
The current llms.txt v2 proposal, updated in August 2026, describes the file as a small guide that agents can view or search before following links to more detailed, ideally LLM-friendly content.
A simplified version might look like:
# Acme Bank
> Official information about Acme Bank products and services.
## Personal Banking
- [Home Loans](/home-loans)
- [Savings Accounts](/savings)
- [Credit Cards](/credit-cards)
## Business Banking
- [Current Accounts](/business/current-accounts)
- [Payments](/business/payments)The intent is useful.
Instead of asking an agent to navigate a large website blindly, the organisation can provide a concise map of the resources it considers useful.
The updated proposal also supports mechanisms for pointing agents towards Markdown versions of individual pages and for associating sections of a site with more specific llms.txt files.
That can be particularly useful for documentation-heavy environments, APIs and other information estates where agents benefit from a guided path through a large body of material.
But this is the crucial distinction:
llms.txt helps organise information. robots.txt communicates crawling policy.
One does not replace the other.
Google makes the difference particularly clear
Google added explicit guidance on llms.txt in June 2026.
Its position is straightforward:
Google Search does not use llms.txt for ranking or visibility, including within its generative AI search experiences.
Creating one will neither help nor hurt a website’s visibility in Google Search. Google says it is perfectly reasonable to maintain an llms.txt file for other services or systems that use it, but it is not required for Google AI Overviews or AI Mode.
Google’s guidance on generative AI optimisation
That is an important reality check.
A brand should not add llms.txt because somebody promised it would improve Google AI rankings.
Google explicitly says it will not.
But that does not mean the concept has no value elsewhere.
The llms.txt ecosystem has continued to develop. The v2 proposal notes growing use across documentation platforms and coding-agent workflows, and some AI companies themselves publish llms.txt indexes for their developer documentation. Perplexity's own documentation, for example, exposes an llms.txt documentation index.
Publishing the file and proving that every general-purpose AI search crawler relies on it are therefore two different things.
Three files, three different jobs
A useful way to remove the confusion is to separate three common website files.
| File | Primary job | What it does not guarantee |
|---|---|---|
| robots.txt | Communicates crawler access preferences | That the content is readable or useful |
| sitemap.xml | Helps search systems discover URLs | That those pages will be indexed, cited or understood |
| llms.txt | Provides a curated, agent-friendly guide to useful content | That AI crawlers will use it or that linked pages are readable |
The distinction matters because organisations sometimes treat all three as different versions of the same machine-readability file.
They are not.
A site can have a perfect sitemap and still block an AI search crawler.
It can allow the crawler and still return an empty JavaScript shell.
It can publish an excellent llms.txt file whose links ultimately lead to pages that machines struggle to process.
Each layer solves a different problem.
llms.txt does not fix the page it points to
This is the most important practical limitation.
Imagine llms.txt tells an agent:
The current account product information is available at /business/current-account.
Useful.
But when the agent requests that URL, the server returns:
<div id="app"></div>
<script src="/app.js"></script>The index was clear.
The page was not.
llms.txt does not execute the JavaScript, reconstruct the page or change the HTTP response the crawler receives.
It points to content.
The content still has to survive the fetch.
This is the same machine-readability problem we explored in AI Crawlability: Why a Fast Website Can Still Be Invisible to AI.
For websites where client-side rendering or heavy page chrome creates that gap, the solution needs to happen at the page or delivery layer, not only in an index file.
Publive AXP takes that approach with AXP Edge’s AI crawler optimisation layer, which serves relevant AI crawlers a clean representation of the existing approved page content while leaving the human-facing website unchanged.
robots.txt does not fix the page either
The reverse is also true.
Suppose the site contains:
User-agent: OAI-SearchBot
Allow: /That may tell the crawler it is permitted to fetch the website.
It says nothing about what the crawler will receive.
The page can still:
- Depend heavily on client-side rendering
- Hide important content behind interaction
- Return the wrong status code
- Be blocked by a CDN or WAF
- Contain stale or contradictory information
- Surround a small amount of useful content with excessive boilerplate
robots.txt controls a gate.
It does not improve what sits behind the gate.
So should enterprise websites publish llms.txt?
In many cases, yes, as a supplementary layer.
The implementation is lightweight, the format is simple, and the 2026 version of the proposal gives agents a clearer way to discover it and navigate towards relevant machine-friendly material.
It can be particularly useful when a site has:
- Extensive technical documentation
- Large knowledge or support centres
- API references
- Complex product catalogues
- Clearly defined authoritative resources
- Markdown versions of important content
But the order of operations matters.
An enterprise should not treat llms.txt as the foundation of AI visibility while more fundamental problems remain unresolved.
Before relying on it, make sure:
The crawlers you want are actually allowed
Review the current documentation for the AI systems relevant to the organisation and configure robots.txt, CDN and WAF policies accordingly.
Important pages are machine-readable
Check what those systems receive when they request the URL, not merely what a person sees after the browser finishes rendering it.
The links point to current information
A curated llms.txt file that directs agents towards obsolete documents can make discovery easier without making the answer more accurate.
The file is maintained
If the website changes while llms.txt stays frozen, the supposedly curated guide eventually becomes another stale source.
That turns a useful navigation layer into another governance problem.
The wrong question is “llms.txt or robots.txt?”
You may need both.
You may also need a sitemap.
And none of those files replaces the work of making the actual pages accessible, current and useful.
Think of the layers this way:
robots.txt: Are you allowed to fetch this?
llms.txt: Here is where useful information lives.
The webpage: Here is the information itself.
That final layer remains the most important one.
Because an AI system cannot cite a crawler policy.
And it cannot answer a customer’s question using an index file alone.
It ultimately needs the underlying information.
In 2026, get the fundamentals in the right order
llms.txt is an interesting and increasingly used agent-discovery convention.
robots.txt remains the established mechanism documented by major AI platforms for expressing crawler-access preferences.
They should not be positioned as competing alternatives.
For enterprise brands, the stronger sequence is:
Control access → make the page readable → guide discovery → keep the information current → measure what AI actually uses
If adding llms.txt helps agents navigate the site, add it.
But do not confuse the presence of the file with AI readiness.
The file can point the way.
The page still has to deliver the answer.
