Your robots.txt Might Be Blocking AI Crawlers

Here’s a thing that sounds like it should be obvious but rarely gets checked: AI platforms send their own web crawlers to index content. A lot of websites are blocking them, and almost none of it is on purpose.

Robots.txt files written years ago, or generated by default CMS settings, often contain blanket disallow rules that catch crawlers nobody was thinking about when the file was last touched. OpenAI’s GPTBot. Anthropic’s ClaudeBot. Perplexity’s PerplexityBot. Google’s Google-Extended. Microsoft’s Bingbot, which now also feeds Copilot. Most of these didn’t exist when the robots.txt file was last reviewed.

The consequence is direct. If a crawler can’t access your content, your content can’t be cited in that platform’s answers. You can publish excellent, authoritative material and still be invisible in AI-generated answers, simply because of a configuration file nobody has opened in three years.

What we found checking this at scale

We ran this check across 84 sites we’ve audited. For each one, we read the actual robots.txt and tested whether it granted, restricted, or fully blocked the crawlers behind the major AI platforms.

Crawler	Powers	Blocked or restricted	Clean access
GPTBot	ChatGPT (training)	21.4%	60.7%
ClaudeBot	Claude	23.8%	58.3%
PerplexityBot	Perplexity	19.0%	63.1%
Google-Extended	Google AI Overview, AI Mode, Gemini	22.6%	59.5%
Bingbot	Microsoft Copilot	17.9%	64.3%

Roughly 1 in 5 sites restrict or fully block at least one major AI crawler. The remainder of each row is sites where we couldn’t confirm status at scan time, not sites that explicitly allow access, so the real number of brands leaving visibility on the table is likely higher than what’s in this table.

Separately, of the sites that ran our free AI Readiness Check, only 1 in 43 scored “AI Ready.” Most landed at “Minimal Presence” or “Basic Web Presence,” the tiers where a crawler either can’t reach the content or can’t make sense of it once it’s there.

Check your own file in five minutes

Open your robots.txt file, it’s at yourdomain.com/robots.txt, and look for any User-agent rules with Disallow directives. If you see a blanket rule like User-agent: * / Disallow: /, you’re blocking everything, including every AI crawler you want to let in.

The fix is explicit. Add specific allow rules for the crawlers in the table above, or review whether your general crawl restrictions are tighter than they need to be. If you’re intentionally blocking certain directories from search engines, that’s fine, just confirm you’re not blocking pages you actually want AI to find.

The layer robots.txt doesn’t show you

Even a clean robots.txt doesn’t guarantee access. Some sites block crawlers at the CDN or server level, in configuration that never shows up in the robots.txt file at all. If your pages load fine in a browser but you’re still not showing up in AI responses despite good content, that’s the next place to look.

For retrieval-based platforms like Perplexity, which pull live web content at query time, this matters even more directly than it does for the training-focused crawlers. Indexing delay becomes citation delay. The faster your content is actually reachable, the faster it can show up in what Perplexity tells your buyers.

Two layers, most teams only check one

A complete AI visibility audit covers the content layer, what you’re publishing and how it’s structured, and the technical layer, whether crawlers can actually reach it. Most teams do the first and skip the second entirely, which is how a well-written page ends up invisible for reasons that have nothing to do with the writing.

beket.ai checks both. We identify technical barriers that may be keeping AI platforms from indexing your content, alongside auditing what ChatGPT, Perplexity, Google AI Overview, Google AI Mode, and Copilot actually say about you once they can see it. Run a free check at beket.ai/ai-check and see what’s blocking your AI visibility.