AI Crawlability: Why a Fast Website Can Still Be Invisible to AI

A website can load quickly, pass Core Web Vitals and offer an excellent experience to customers.

An AI crawler may still receive almost none of its important content.

There is no contradiction here.

Google’s Core Web Vitals measure user-centred outcomes such as loading performance, responsiveness and visual stability. Google’s Core Web Vitals guidance

These are important indicators of website quality. But they do not tell you whether a machine fetching the page receives the same product information, FAQs, pricing, eligibility criteria and supporting evidence that a customer eventually sees in a browser.

This creates a growing blind spot for enterprise teams.

The website appears healthy through conventional performance and SEO tools. A customer opens the page and sees everything expected. But the initial response from the server may contain little more than scripts, navigation and an empty application shell.

For AI discovery, that difference matters.

The issue is not simply whether a page is fast.

It is whether the information that matters is available when a machine asks for it.

What is AI crawlability?

AI crawlability is the ability of an AI-related crawler or fetcher to reach a page and access enough useful information to process what that page is about.

That means asking:

• Can the crawler reach the URL?
• Is it permitted to access the content?
• What does the server actually return?
• Is the page’s important information present?
• Can the machine identify the relevant facts and context?
• Can it follow important links?
• Is the information current and internally consistent?

Traditional crawlability has largely focused on whether search engines can discover and index a page.

AI crawlability adds another question:

What does the machine actually receive after it reaches the page?

A crawler may be allowed through robots.txt but still receive an incomplete response.

A URL may return 200 OK while much of its substantive content appears only after client-side JavaScript executes.

A firewall may allow ordinary browsers while challenging or blocking automated requests.

Access is necessary.

It is not the same as readability.

Why a strong performance score does not prove AI readability

This is where many enterprise websites can develop a machine-readability gap.

A customer may see a complete product page, while an automated system receives a thinner initial response and depends on JavaScript to reconstruct the rest.

Google itself distinguishes between crawling, rendering and indexing when processing JavaScript websites. Its documentation explains that some applications use an app-shell model, where the original HTML contains very little actual page content and JavaScript generates the rest later. Google’s JavaScript SEO documentation

Googlebot can render JavaScript through Chromium.

But Google also notes that not all bots can run JavaScript.

So a fast browser experience does not prove that every machine accessing the same URL receives the same information.

Raw HTML and the rendered page are not the same thing

Consider an insurance product page.

A customer eventually sees:

• Policy name
• Coverage limits
• Premium information
• Eligibility criteria
• Exclusions
• Network hospital information
• FAQs

But the initial HTML response might contain only:

<div id="app"></div>
<script src="/assets/product-page.js"></script>

The browser executes the script, retrieves additional data and builds the finished page.

The customer sees a complete experience.

A machine processing only the initial response sees an empty container and a JavaScript file.

Now compare that with a server response where important information is already present:

<h1>Business Current Account</h1>

<p>
  Available to registered businesses with digital onboarding
  and no minimum opening deposit.
</p>

<h2>Account charges</h2>

The second response gives the machine immediate information about the entity, product and key attributes.

The conclusion is not that JavaScript is bad.

It is that organisations should not assume every machine processes JavaScript in the same way, waits for the same events or receives the same final DOM as a customer’s browser.

Not every AI crawler has the same job

“AI crawler” is also not one category.

Different automated agents may access websites for search discovery, user-directed retrieval, model development or other AI-assisted experiences.

The distinction matters because platforms expose different controls for different use cases.

OpenAI

OpenAI distinguishes between OAI-SearchBot and GPTBot.

OAI-SearchBot is relevant to discovering and surfacing websites in ChatGPT search. OpenAI advises publishers that want their content included in ChatGPT summaries and snippets not to block it. GPTBot relates to potential model training and can be controlled separately. OpenAI’s publisher and crawler guidance

OpenAI also advises website operators to ensure their host or CDN permits traffic from its published SearchBot IP ranges.

Anthropic

Anthropic similarly distinguishes between ClaudeBot, Claude-SearchBot and Claude-User.

Claude-SearchBot supports search, while Claude-User can retrieve pages in response to a user request. Anthropic’s crawler documentation

Perplexity

Perplexity documents PerplexityBot for search discovery and Perplexity-User for user-directed retrieval.

Its documentation also advises organisations using a web application firewall, or WAF, to validate both the user agent and published IP ranges when configuring access. Perplexity’s crawler documentation

The business implication is simple:

An enterprise needs to know which machines it wants to serve, what purpose they perform and what those systems actually receive.

Where machine readability commonly breaks

Client-side rendering

In a client-side rendered application, the server may return an application shell while the browser constructs the substantive page later.

If the crawler does not complete the same rendering process, important content can be absent.

Hydration and delayed API calls

A page may contain some server-rendered HTML but depend on hydration, where JavaScript activates and completes the page in the browser, or on later API calls before important information appears.

That could include pricing, eligibility, interest rates, inventory or location availability.

If those later requests fail, time out or are not executed by the crawler, the machine may receive only part of the page.

Interactive and lazy-loaded content

Important information is also frequently placed inside:

• Accordions
• Tabs
• Calculators
• Comparison tools
• Expandable FAQs
• Scroll-triggered sections

Lazy loading itself is not a problem.

The risk appears when understanding a product depends on an interaction the machine does not perform.

CDN, WAF and bot controls

Sometimes the content is perfectly readable but the crawler never reaches it.

A website may allow a crawler in robots.txt while its CDN or web application firewall blocks the request further upstream.

Instead of the actual page, the crawler may receive:

• 403 Forbidden
• A rate-limit response
• A CAPTCHA
• A JavaScript challenge
• A consent screen
• A generic fallback page

Perplexity’s own documentation, for example, provides WAF configuration guidance and recommends combining user-agent checks with official IP ranges.

This matters because AI visibility is not only an SEO or content issue.

Infrastructure and security teams can unintentionally block the same systems marketing wants to reach.

Incorrect status codes and redirects

Machines also rely on HTTP status codes to understand what happened when a page was requested.

Google recommends returning meaningful status codes and specifically warns about soft 404s in client-rendered applications.

A page that visually says “not found” but returns 200 OK sends an ambiguous machine signal.

This is not just a theoretical problem

Publive AXP and IAMAI examined 951 enterprise and mid-market websites for the India AI Visibility Benchmark 2026.

The research found that 94% failed the foundational requirements used in the benchmark for AI readability. It also found that 48 brands served AI crawlers no usable content, while for the median site, 84% of what an AI bot downloaded consisted of code and markup rather than content.

Those findings reveal an important distinction.

A page can exist.

It can rank.

It can look excellent.

And the useful information available to another machine can still be dramatically different from what the customer sees.

Crawlability should also include source integrity

Making content easier to retrieve does not automatically make the resulting answer correct.

Imagine improving the machine readability of a current product page while leaving an outdated brochure publicly accessible.

The current page says a service is available in 20 cities.

The older PDF says 12.

The AI can now read both perfectly.

The technical problem has been solved.

The information problem has not.

An AI crawlability review should therefore also ask:

• Is this the current version of the information?
• Are superseded pages redirected or clearly archived?
• Are outdated PDFs still discoverable?
• Does structured data agree with the visible page?
• Do regional and product variations contain the right qualifiers?

Machine readability and information governance need to support each other.

The goal is not to make every public source equally retrievable.

It is to make the correct, current and properly qualified source easy to find and understand.

This is where crawlability connects to the wider brand-accuracy problem explored in Publive AXP’s Your Brand Is Visible in AI. But Is It Accurate?

How should enterprises test AI crawlability?

Start by comparing what the server sends with what the customer eventually sees.

Inspect the initial response

Fetch a priority page without relying on full browser-side execution.

curl -L -s https://www.example.com/product-page > raw-page.html

Then look for an actual product fact, price, eligibility condition, FAQ answer or important claim.

Do not check only whether the title exists.

Compare it with the rendered page

Open the same URL normally and compare:

• Main copy
• Product attributes
• Tables
• FAQs
• Important links
• Structured data
• Canonical information

Google recommends inspecting rendered HTML when diagnosing JavaScript-related search problems. Google’s JavaScript troubleshooting guidance

Review crawler access

Check the access layer separately:

• robots.txt
• HTTP headers
• CDN rules
• WAF policies
• Rate limits
• CAPTCHA behaviour
• Authentication or consent dependencies

And test the specific crawler relevant to the objective.

For ChatGPT search discovery, for example, that means checking OAI-SearchBot, not simply GPTBot.

Inspect server and CDN logs

Logs can show:

• Which URLs were requested
• Which crawler requested them
• Response status
• Response size
• Redirect behaviour
• Failed requests

That moves the conversation from:

“We think the crawler can access us.”

to:

“We know what it received.”

Test representative page templates

A homepage test is not enough.

Include product pages, category pages, articles, location pages, campaign pages and pages containing important commercial or legal claims.

A rendering decision at the template level can affect thousands of URLs.

What should enterprises fix?

There is no single technical implementation every website needs.

For websites where architecture changes are practical, server-side rendering, static rendering or hydration can reduce dependence on client-side execution for important content.

Google recommends these as stronger long-term approaches than dynamic rendering. Google’s guidance on dynamic rendering

The broader principles are more important than the framework chosen:

• Keep essential information available in meaningful HTML.
• Use clear headings and crawlable links.
• Return accurate status codes.
• Avoid hiding material facts entirely behind interactions.
• Ensure security layers allow the crawlers the organisation intends to support.
• Keep structured data consistent with visible content.
• Verify machine access rather than assuming it.

For large existing platforms, however, rebuilding the entire rendering architecture solely for AI crawlers may not be practical.

An infrastructure layer can provide another route by improving the representation delivered to relevant AI agents while leaving the human-facing website unchanged.

Publive’s AXP Edge AI Crawler Optimization layer takes this approach by serving a clean, structured representation of existing approved page content to relevant AI crawlers at the edge.

The important principle is factual equivalence.

The machine-facing representation should make existing information easier to access and process. It should not invent or materially alter what the organisation has published.

Fast for people and readable by machines

Website speed remains important.

Brands should continue investing in Core Web Vitals, user experience and technical SEO.

But these measures answer a different question.

Performance asks: How well does the page work for a person?

AI crawlability asks: What does an automated system receive, and can it extract the information that matters?

Enterprises increasingly need to measure both.

Because before an AI system can understand a brand, cite its evidence or recommend its products, it first needs a usable representation of what the brand has published.

Crawlability is only the first layer:

Access → Understand → Cite → Recommend → Refer

If the information does not survive the first step, none of the later ones can happen.