AI Crawler Accessibility Test

Why AI Crawler Accessibility Matters in 2026

Section 1 — The AI citation gap: invisible sites lose traffic they cannot see

AI assistants have become a primary research channel for software buyers. According to the Previsible AI Traffic Report (July 2026, 6.77 million sessions), ChatGPT alone accounts for 92.4% of measurable LLM referral traffic to websites — and visitors referred by AI assistants convert at approximately 18%, the highest measured conversion rate of any traffic source in the study.

Source: Previsible, "AI Traffic Report", July 2026, 6.77M sessions analysed.

A site blocked to GPTBot, ClaudeBot, or PerplexityBot cannot appear in AI-generated answers, regardless of its content quality. The exclusion is silent — no error, no warning. The site simply never appears as a citation source. The tool above checks for this gap in seconds.

Section 2 — The four AI crawlers and their User-agent strings

These are the canonical, verifiable User-agent strings published in each provider's official documentation. All four documentation URLs are live and publicly accessible.

Section 3 — What blocking these crawlers actually does

A Disallow: / directive for any of these User-agents tells that AI platform's crawler it cannot access any page on the site. The AI has no data to cite and the site never appears in AI-generated answers for any query — even queries where the site is the most relevant source on the web.

The most common blocking scenario is not intentional: site owners add a User-agent: * / Disallow: / block to prevent generic scrapers from harvesting content, without realising that the wildcard rule also blocks GPTBot, ClaudeBot, and every other AI crawler that has no explicit Allow rule. This tool surfaces that gap without requiring any account or sign-up.

Section 4 — How to allow AI crawlers (copy-paste snippet)

Add the following blocks to your robots.txt file. Place them before any User-agent: * wildcard block so the explicit rules take precedence:

# Allow all major AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Note: Allow: / is only needed if a preceding User-agent: * block contains a Disallow: /. If your robots.txt has no wildcard block, or your wildcard block does not disallow root, these crawlers are already accessible by default and no explicit entry is needed. The robots.txt default is allow per the published Google specification.

Section 5 — llms.txt: the next layer of AI discoverability

llms.txt is an emerging web standard — modelled on robots.txt — that provides AI agents with a structured, machine-readable index of a site's most important pages and capabilities. It is hosted at https://yourdomain.com/llms.txt and uses plain Markdown-style formatting so AI assistants performing web-grounded retrieval can identify the site's key pages without crawling the entire domain.

The format was initiated by Jeremy Howard (fast.ai) in 2024 and is documented at llmstxt.org. It does not yet have an official IETF standard (accurate as of 2026). Having an llms.txt is a positive AEO signal; its absence is not a penalty — AI crawlers will still index the site via normal robots.txt rules. The tool above checks whether your domain currently has one.