More than an llms.txt file: serve AI crawlers a clean view of every page
AI crawlers don’t run your JavaScript. AXP Edge renders and cleans your pages at the CDN edge, so bots read 100% of your content in under 500ms. No rebuild.
AI crawlers fetch your HTML. They don’t run your JavaScript.
Modern sites ship a near-empty document and paint the real content with JavaScript in the browser. Human browsers execute it. Most AI crawlers do not.
<div id="root"></div> <script src="/app.[hash].js"> <!-- product facts load client-side, after JS --> // crawler sees: (empty)
<h1>Acme Health Cover</h1> <p>Cashless at 12,000+ hospitals. ₹5L–₹1Cr cover.</p> <ul><li>98.6% claim ratio</li> </ul> // crawler sees: everything
This is why a page that ranks perfectly well on Google can be invisible inside AI answers: Googlebot has rendered JavaScript for years, but the agentic crawlers behind ChatGPT, Claude and Perplexity mostly fetch raw HTML and move on. If your facts aren’t in that first response, they aren’t in the model’s answer.
One request in. The right page out for every visitor.
AXP Edge sits on your CDN, spots who is asking, and routes AI bots to a clean, agent-first version while humans and search engines see your site exactly as it is today.
What is llms.txt, and why it isn’t enough
llms.txt is a proposed convention: a plain-text file at the root of your domain — yourdomain.com/llms.txt — that lists the pages, in a Markdown-friendly form, you most want large language models to read. The idea is sound and the intent is right: give AI a clean, prioritised map of your content instead of making it crawl the whole site blind. If you publish documentation or a large catalogue, an llms.txt file is a reasonable thing to add.
But it is worth being honest about what the file actually does. llms.txt is a pointer. It tells a model where your important content lives; it does nothing to change what the model receives when it fetches those URLs. If a listed page renders its content with client-side JavaScript, the crawler that follows the link still gets the same empty shell it always did. The index improved; the pages did not. And because llms.txt is a community proposal rather than an adopted standard, no major AI crawler is documented to fetch or act on it today — so treating it as your AI-visibility strategy is a bet on future behaviour, not present reality.
What llms.txt does
What it does not
The unit that matters is the page, not the index. When someone asks an AI tool a question, the model reaches for a specific, relevant page and reads whatever HTML comes back. A single root-level file can’t make thousands of those responses clean, structured and citation-ready. Per-page rendering can — which is the job AXP Edge does at the edge, for every URL, whether or not you ever publish an llms.txt file.
The bots reading you right now
They arrive as identifiable user-agents, request your URL over HTTP, and read the bytes that come back. Most stop there — no browser, no JavaScript engine, no second pass.
An AI crawler is, mechanically, a simple client. It sends a GET request with a distinctive User-Agent string, receives your HTML, and extracts text. The crucial difference from a real browser is that it typically does not execute JavaScript. There is no render step, no hydration, no waiting for fetch() calls to resolve. Whatever is in the raw HTTP response is all the crawler will ever see. A fetch-survivable page is one whose meaning is fully present in that first response: real headings, readable paragraphs, lists, tables and structured data — no dependence on a client-side render to become legible.
| Company | User-agents | What it’s for |
|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | Powers ChatGPT browsing, search and training fetches. |
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot | Fetches pages to ground and cite Claude’s answers. |
| Perplexity | PerplexityBot | Retrieves sources for Perplexity’s cited answers. |
| Google-Extended | Controls use of your content for Gemini and AI features. |
You can allow or block any of these from robots.txt, and Google-Extended in particular is a policy signal for whether your content trains and grounds Gemini. But blocking is a blunt instrument: keep the bots out and you keep yourself out of the answers. The higher-leverage move is to let them in and make sure they can read you — which requires the page itself to be fetch-survivable.
Three ways to make a page machine-readable
There is more than one way to get real content into the first HTTP response. Server-side rendering (SSR) builds the HTML on your application server for every request — robust, but it usually means adopting or migrating to an SSR framework and taking on the server cost. Prerendering (static generation) bakes HTML at build time — fast and cheap, but awkward for large or frequently-changing catalogues, and it still serves crawlers the same human-oriented markup. Edge rendering, the AXP Edge approach, runs at the CDN in front of your existing site: it renders the page, then serves AI crawlers a cleaned, token-lean representation — without touching your application, your origin or what humans and Googlebot receive.
| SSR | Prerendering | Edge rendering (AXP Edge) | |
|---|---|---|---|
| Where it runs | Your application server / build | Your build pipeline | Your CDN edge, per request |
| Needs an app rebuild | Often (framework migration) | Build-time integration | No — installs as a worker |
| Serves AI a clean, token-lean view | Same HTML as humans | Same HTML as humans | Purpose-built agent HTML |
| Impact on human page-speed | Can add server latency | Neutral | Zero — humans hit origin |
| Rollback | Redeploy | Rebuild & redeploy | One click |
SSR and prerendering are excellent choices when you’re building or rebuilding. Edge rendering exists for the far more common case: a site that already works, that you don’t want to re-architect just to be legible to AI. It’s additive, reversible and invisible to everyone except the crawlers it’s meant for.
A worker on your CDN, not a rebuild of your app
One small install on the edge you already run — Cloudflare, CloudFront or Akamai — live in under two hours, with failover and one-click rollback built in.
Detect
A lightweight worker on your CDN inspects each request. AI crawlers are identified by user agent, matched against a list we maintain; humans and search bots are passed straight through.
Render & clean
For a matched crawler, the edge renders the page and strips it to substance — no nav, scripts or chrome — emitting clean, semantically chunked, schema-rich HTML in under 500ms.
Fail safe
The optimised view is an enhancement, never a dependency. If the edge layer is slow or unavailable, the request falls through to your normal origin — no error a visitor wouldn’t otherwise have seen.
Roll back
Turning it off is one click. Because nothing in your application changed, removing the worker returns every request to exactly today’s behaviour, instantly.
Runs on the edge network you already have
Cloudflare
Deployed as a Cloudflare Worker in front of your zone.
Amazon CloudFront
Runs as a Lambda@Edge / CloudFront Function on your distribution.
Akamai
Installs as an EdgeWorker on your existing property.
Serve your site through another CDN or edge network? If it can run edge logic in front of your origin, we can almost certainly integrate — talk to us about your setup.
One web address. Two audiences.
Flip the toggle to see what actually leaves the edge. Same substance, served the way each audience needs it.
Health insurance that keeps up with your family
Cashless treatment at 12,000+ hospitals. Cover from ₹5 lakh to ₹1 crore.
Only AI gets the clean view
The foundation, rebuilt at the edge
Every number here traces back to one thing: AI reading a clean, structured version of the page you already have.
AI visibility, built on infrastructure already proven at enterprise scale
/thepublive/media/media_files/2026/04/06/untitled-design-3-2026-04-06-15-28-38.png)
/thepublive/media/media_files/2026/03/11/69a694b9788b8cc26350f8c4_finopaymnetbank-20-1500536275-2026-03-11-17-33-55.jpg)
/thepublive/media/media_files/2026/03/11/afaqs-2026-03-11-16-20-44.webp)
/thepublive/media/media_files/2026/03/11/sanjeev-kapoor-2026-03-11-16-20-44.webp)
/thepublive/media/media_files/2026/03/11/elle-2026-03-11-16-20-45.png)
Two things start to move in the first few weeks
AI agents fetching your pages, and AI tools sending real visitors back to you.
| Page | Agent traffic | AI citations | AI referrals |
|---|---|---|---|
Publive | AI Visibility Platform publive.online/ | 6,240+34% | 41+18% | 318+27% |
Publive | AXP Edge /axp-edge | 2,180+52% | 12+60% | 96+44% |
Publive | Streams /streams | 1,940+21% | 23+15% | 142+12% |
Publive | Blog: the GEO playbook /blog/geo-playbook | 1,360+28% | 9+12% | 71+9% |
Publive | CMS /cms | 980+15% | 6flat | 38+6% |
Enterprise-safe by design
Nothing changes for customers
Only AI tools get the cleaned-up version. Real visitors and search engines see today’s site, untouched.
This is not cloaking
We only clean up and restructure what is already on your page, AI and visitors get the same substance.
It stays on your own domain
No second page to manage. AI reads your real web address, so citations point straight back to you.
It cannot break your site
If anything ever goes wrong on our side, the page falls straight back to your normal site.
Once crawlers can read you, get cited
AI Search Optimization
Optimize for ChatGPT, Perplexity, Gemini and Google AI Overviews.
Generative Engine Optimization
GEO without a rebuild — earn citations in generated answers.
AXP Edge
The full product: get found in AI answers without changing your site.
AI Streams
Publish new, governed content built to be cited — the add-new-content complement.
Pricing that scales with your AI visibility
Prices exclude applicable taxes. Yearly billing saves 20%. Switch currency and billing above.
llms.txt, crawlers and the edge — answered
llms.txt is a proposed plain-text file at the root of a domain (yourdomain.com/llms.txt) that lists Markdown-friendly links to the pages you most want language models to read. Think of it as a hint file, similar in spirit to robots.txt or sitemap.xml, but aimed at LLMs. It is a community proposal, not an official standard, and no major AI crawler is documented to fetch or act on it today.
Start being the answer today
Tell us where you want to show up in AI answers. We’ll come back with your AI Visibility Gap Report and the fastest path to being cited.