What AI Crawlers Can Actually See on Your Site

Most “AI visibility audit” content jumps straight to accuracy: is the AI saying the right thing about you? That question only matters once you know the AI can actually read your site in the first place — and a surprising number of businesses have never checked.

The access question comes first

Every major AI crawler respects (or ignores) robots.txt differently, and most sites configure it once, for search engines, years before AI crawlers existed. That configuration often blocks or restricts AI bots without anyone deciding to.

Across the sites we’ve checked, here’s how access actually breaks down by crawler:

AI crawler	Allowed	Blocked	Restricted (partial)	Unclear
GPTBot (OpenAI)	60.7%	4.8%	16.7%	17.9%
ClaudeBot (Anthropic)	58.3%	7.1%	16.7%	17.9%
PerplexityBot	63.1%	2.4%	16.7%	17.9%
Google-Extended	59.5%	6.0%	16.7%	17.9%
Bingbot (Copilot)	64.3%	1.2%	16.7%	17.9%

Nearly a fifth of sites don’t even have a clear, resolvable rule for a given crawler — not blocked, not allowed, just ambiguous enough that different crawlers interpret it differently.

The risk isn’t just “invisible”

Being blocked is the obvious problem. The less obvious one is being wide open on pages you didn’t mean to expose — internal search results, staging subdomains, or account-gated pages that got left crawlable because nobody audited the rule since it was written.

Of the free AI-readiness checks run through our tool, only about 1 in 43 came back as fully “AI Ready” with clean, deliberate access rules. The rest ranged from minimal presence to partially accessible with gaps nobody had reviewed.

What to actually check

• Which AI crawlers are named explicitly in your robots.txt, and which are covered only by a wildcard rule that may not do what you assume
• Whether any customer-only or internal paths are crawlable that shouldn’t be
• Whether your rule was last touched before AI crawlers existed at all

Fixing this isn’t about certifications or compliance checkboxes — it’s a five-minute file that most sites haven’t looked at since it was written for search engines, not AI models.

beket.ai’s free Robots.txt Checker shows you exactly how each major AI crawler reads your current rules, so you know what’s actually visible before you worry about what it says.