How to Make a Website Readable to AI Crawlers

Most questions about AI crawlability arrive as an implementation question: can we serve an AI-optimised version of our site to AI bots through Cloudflare, Akamai or CloudFront? The short answer is yes. All three can select a response per request at the edge. Whether you should depends on three things you can test before anyone writes code: what each crawler receives today, which delivery path you actually control, and whether a bot-specific response stays faithful to what users and search engines see.

This guide works through those three steps in order. Every test in it names the crawler, the delivery path and the version being tested, because “AI can’t read our site” is not a finding. “OAI-SearchBot receives the page title and navigation but no price table on /pricing, served by CloudFront from the origin, release 2026.11.2” is a finding, and it tells you what to fix.

Audit what crawlers can access

Inspecting what AI systems can read from your site starts with what they say about it. Brand AI Observability tracks the prompts that matter to your buyers across ChatGPT, Gemini, Claude and Google AI Overviews, and records the sources each answer cites. When a priority page never appears among those sources, or an answer repeats an old price or a discontinued product, that is the signal to look at what the page delivers to machines. Observability shows where the gap appears. The audit below shows why.

Why can’t ChatGPT and other AI bots read my website’s JavaScript content?

Many AI crawlers fetch the HTML a server returns and do not reliably run the JavaScript that builds the page in a browser. If the main copy, prices or product claims are injected client-side, those crawlers can receive an empty shell or a loading state. Behaviour differs by crawler and changes over time, so test each one rather than assuming every AI bot renders pages the way Googlebot does.

Google does render JavaScript for Search, though it describes rendering as a separate stage after crawling (JavaScript SEO basics). Practitioners are less certain about the rest. Harry Sanders put the question to his LinkedIn audience directly: of the AI crawlers you’ve tested, do any of them actually execute JS, or is plain HTML the only safe bet right now? His working rule is a good one to audit against: if the main service, price or key claims only appear after JavaScript runs, move that copy into the server-rendered HTML.

Step 1: Check robots.txt and the controls robots.txt does not cover

Fetch /robots.txt and write down the rules that apply to each named agent. OpenAI’s crawler documentation separates GPTBot, used for model training, from OAI-SearchBot, used for ChatGPT search, and from ChatGPT-User, which fetches pages when a person asks ChatGPT to. Each is controlled separately. Disallowing GPTBot opts out of training, not out of ChatGPT search, and OpenAI notes that robots.txt rules may not apply to fetches a user initiates.

Then check the layers robots.txt does not show. Meta robots tags and X-Robots-Tag headers can stop use of a page that robots.txt allows. WAF rules, bot-management settings, rate limits and country blocks on the CDN can turn a crawler away before the origin sees the request. A rule that returns a 403 to unrecognised bots blocks a crawler as completely as a Disallow line, and it will never appear in robots.txt.

Step 2: Capture the plain HTML response

Request each priority URL with the crawler’s user-agent string and save what comes back before any script runs: the status code, the redirect chain, the response headers and the body. Take the current string for each crawler from its operator’s documentation. A plain command-line request is enough:

UA="<current OAI-SearchBot string from OpenAI's crawler docs>"
curl -sS -L -D oai-searchbot.headers -A "$UA" \
  https://www.example.com/pricing -o oai-searchbot.html

Know the limit of this test. A request sent from your own network with a borrowed user-agent shows what your stack returns to that string. It does not prove what the real crawler received, because bot managers can verify genuine crawlers by IP range or signed requests and treat an imitation differently. Confirm the result against verified requests in your server or CDN logs; our guide to identifying AI crawlers in server logs covers how.

Checking tools run the same test from someone else’s network. Santosh Subramanian’s round-up of crawlability checkers describes the llmrefs AI crawlability checker as one that “uses ChatGPT’s official GPTBot user agent to show you the exact content AI systems can read.” That is useful, with the same caveat as your own request. Our free AI Content Visibility Grader compares the raw HTML available to an AI fetch with the fully rendered page a person sees.

Step 3: Compare it with what users see after JavaScript runs

Load the same URL in a headless browser, let the page settle, and save the rendered DOM. Compare the two versions on the elements that carry meaning: the H1 and headings, body copy, prices, plans and eligibility, product names, tables, internal links, the canonical tag and structured data.

Put every element that is present after rendering but missing from the raw HTML under one label: Claims only appear after JavaScript. That list is the main output of the audit. It is specific to named facts, URLs and crawlers, and every item on it can be fixed. Record each test in the same shape so results stay comparable across releases:

Choose an implementation path

AXP Edge is Publive AXP’s edge-delivery option for agent-facing content. It installs as one edge function in the CDN account you already run, as a Cloudflare Worker, a CloudFront function or an Akamai EdgeWorker, and serves recognised AI agents a clean, rendered view of the existing page while your application, CMS and URLs stay as they are. It is one of four paths, and the audit decides which one fits.

Should we use a CDN edge worker to serve AI crawlers an optimised version of our pages, or build it in-house?

Fix the application when your team controls the framework and can move the missing content into server-rendered HTML for every visitor, because one response then serves people, search engines and AI crawlers alike. Choose an edge path when the CMS or front end cannot change quickly, or when the gap spans many templates. Build the edge layer in-house only if you can also own crawler identification, caching, rendering, monitoring and failover for as long as it runs.

Our comparison of edge rendering, SSR and prerendering goes deeper on the architecture behind each row.

How to optimise an Adobe AEM or Next.js website for AI answer engines

On Next.js the application path is usually the cheapest. The framework supports server rendering and static generation, so when the audit finds key copy inside components that fetch their data in the browser after the page loads, the fix is to render that content on the server instead.

Adobe Experience Manager sites typically deliver server-rendered HTML, but single-page-application implementations, client-side personalisation and content fragments loaded by script can still move claims out of the first response. Changes there pass through authoring workflows and release governance that can take weeks. That is where an edge path earns its place, because it works on the delivered page without waiting for the next platform release.

Delivery only moves what the page already contains. Ahrefs’ course on AI search turns from what AI systems choose to how to actually structure your content so AI can use it, and that layer sits on top of whichever path you pick: descriptive headings, the answer stated before its qualifications, tables where a comparison needs one. Our guide to AI content optimisation covers the editorial side.

Can I serve an AI-optimised version of my site to AI bots through Cloudflare, Akamai or CloudFront?

Yes. All three can inspect a request at the edge and select a different response for identified AI crawlers while real users keep the normal page. The safe pattern is the same on each: identify verified crawlers, keep the same URL with no redirect, leave the human and search-engine response unchanged, keep the two versions apart in cache, and fall back to the origin page on any error.

Here is how the selection works on each platform:

• Cloudflare: a Worker on the route inspects each request, and Cloudflare’s bot signals can tell a verified crawler from an imitation. Cloudflare also offers Markdown for Agents, which converts the origin’s HTML to Markdown when a client asks for it. That changes the format of what the origin already returns. It does not add content that was missing from the HTML.
• Akamai: Bot Manager classifies the request and an EdgeWorker or a Property Manager rule selects the response or origin. Akamai describes pairing bot classification with edge logic to decide per request what each client receives. In Property Manager the last matching rule wins, so a later rule can quietly override a bot route, and a bot blocked by an earlier security rule never reaches the route at all.
• CloudFront: a CloudFront function or Lambda@Edge on the viewer request selects the response. AWS advises against putting the User-Agent header in the cache key because it has thousands of variations, and its own guide to bot-specific content on CloudFront identifies crawlers with AWS WAF Bot Control instead.

Confirm routing and redirect behaviour before go-live

Whichever CDN you run, write down and test each of these behaviours on staging, then again in production:

1. Same URL, no redirect. The crawler requests /pricing and receives /pricing. A redirect to a separate AI URL creates a new page that can be crawled, indexed and cited on its own, and splits signals from the original.
2. Selection by verified identity. Match on the CDN’s verified-bot classification, not on the user-agent string alone, which anyone can send.
3. Search crawlers stay on the user page. Googlebot and Bingbot keep receiving exactly what users receive.
4. Separate cache entries. The bot response must never be served to a person. Key the cache on the bot classification rather than the raw user-agent.
5. Fail open. A timeout or error on the bot path returns the origin page, marked with a header so it shows up in monitoring.
6. One-step rollback. One change returns all traffic, agents included, to the origin.

For AXP Edge, this is the routing contract on all three CDNs. Requests on your targeted paths that come from recognised AI agents receive the optimised view; every human browser and every search crawler passes through to your origin byte-identical to before. There is no new URL, no redirect and no change to your sitemap, canonical tags or internal linking. If the AXP backend errors, times out or is unreachable, the edge function serves your origin page with a diagnostic header, and removing the route binding returns 100% of traffic to the origin. Go-live is verified with three checks: an agent request carries the AXP header, a human request is byte-identical to before, and a simulated backend failure falls back cleanly to the origin.

Is it safe to route AI bot traffic through a third-party service without downtime or redirects for real users?

It can be, if the design keeps people off the bot path and treats the service as optional. Real users should never be routed through it, no request should be redirected, and any timeout or error from the service should return the origin page. Without that fail-open design, the service becomes a dependency on your critical path.

The risk is not hypothetical. On 18 November 2025, Cloudflare traced a global outage to a Bot Management configuration file that roughly doubled in size; the proxy software that reads it failed, and sites across its network returned HTTP 5xx errors to ordinary visitors (Cloudflare’s postmortem). The lesson is not to avoid edge logic, since your CDN already runs plenty of it. It is that bot classification and edge code sit on the request path for everyone, so a fault there is never confined to bots.

Put three questions to any vendor, including us. What happens to a human request if your service is down? What happens to an AI crawler request? How quickly can we remove the route? For AXP Edge the answers are: nothing, because people never touch the AXP path; the crawler receives your origin page; and one change. Publive’s infrastructure runs at 99.99% uptime across more than 1B pageviews a month, and failover rate is tracked as a dashboard metric.

Validate parity and search impact

Publive AXP exists to make global brands visible, readable and citable across AI-powered search engines and generative AI platforms. Readable is the part that needs proof. A bot-specific response is acceptable only if it says the same thing as the page people see, and only if search engines keep receiving the page they have always received.

Test three audiences against the same URL and version

For each priority URL, capture three responses from the same deployed release in the same test window: a real browser after rendering; a search crawler, using URL Inspection in Search Console for Googlebot; and each AI crawler on the route. Record every material difference in facts, prices, availability, claims, disclaimers, headings, links, canonical tag, structured data, status code and redirects.

Separate presentation from substance. Removing navigation, scripts, cookie banners and tracking is a presentation difference, and it is the point of an agent view. A price, claim, product or condition that is present in one version and absent from another is a material difference, and it blocks the release. In regulated sectors such as banking and insurance, disclaimers and eligibility conditions belong on the substance side of that line.

Run the comparison again on every release. A cached agent view can fall behind the page after a content change, so a price update is only finished when all three columns show the new value.

Will optimising for AI answer engines hurt my Google SEO rankings?

Not if Google keeps receiving the same page your users receive. Google states that appearing in AI Overviews and AI Mode has no additional requirements beyond standard SEO best practice. The risk comes from implementation errors: redirects to new URLs, a bot route that also catches Googlebot, a cache that serves a stripped page to people, or a search crawler blocked by accident.

Google’s AI features documentation sets the eligibility bar: a page must be indexed and eligible to appear in Search with a snippet. Its AI optimisation guide adds that site owners do not need to create machine-readable files, AI text files or Markdown versions to appear in Google’s generative AI search. Google speaks for Google. ChatGPT, Perplexity and other platforms run their own crawlers, which is why the audit tests each one separately.

What is technical GEO and how is it different from technical SEO?

Technical GEO is the set of technical controls that decide whether AI systems can reach, read and extract a site’s content for generated answers. It shares most of its foundations with technical SEO: crawl access, indexability, server-rendered content, status codes and canonicals. It differs in audience and in the test of success: more crawlers with different rendering abilities and robots.txt tokens, judged by whether content is extracted and cited in answers rather than by ranking position.

Similarweb’s guide to technical GEO frames it the same way, as technical SEO expanded for more crawlers and more retrieval systems. In practice that splits the work in two:

• Shared controls, validated once: status codes, canonicals, indexability, internal links, structured data and server-rendered content.
• Separate validation: robots.txt rules per AI crawler, the raw HTML each non-rendering crawler receives, any bot-specific route and its parity, and citation outcomes in AI answers.

Our explainer on GEO vs AEO vs SEO covers how the disciplines relate beyond the technical layer.

Is serving a different version of my website to AI bots considered cloaking or black hat?

Google’s spam policies describe cloaking as presenting different content to users and search engines with the intent to manipulate rankings and mislead users. A bot-specific format is not automatically cloaking, but a version whose substance differs from what users see carries that risk and erodes trust whatever the policy wording. The defensible line is parity: the same facts, claims and conditions, presentation differences only, and search engines receiving the user-facing page.

Google’s spam policies are the reference for Search. Its documentation on dynamic rendering, which served crawlers a pre-rendered page, describes that technique as a workaround rather than a long-term solution, and says it is not treated as cloaking as long as it produces similar content. Similar content is the test worth adopting for every audience, not just Googlebot.

Loganix’s explainer on cloaking raises the question most teams eventually ask: “What about white hat cloaking?” Treat it as a prompt for your review, not as an answer. The phrase is not a search-policy category and not a legal conclusion. What a reviewer can check is whether the bot response is transparent, meaning documented, marked with a response header and reversible in one step, and faithful, meaning it carries the same substance as the page people see.

Run the steps in order: audit what each crawler receives, choose the path that fits your stack, and validate parity on every release. If the audit shows claims only appear after JavaScript and your platform cannot change quickly, see how AXP Edge’s AI crawler optimisation serves verified AI agents a readable page on the CDN you already run.