Free tool

AI robots.txt checker

Check if your robots.txt allows or blocks AI crawlers. It checks 21 bots from OpenAI, Anthropic, Google, Perplexity, and others. Instant results, no signup.

21 bots checked ~3 sec Free · No signup

FAQ

Questions about robots.txt and AI crawlers

What is robots.txt and why does it matter for AI?: robots.txt is a plain-text file at the root of your website that tells crawlers what they're allowed to access. AI companies like OpenAI, Anthropic, and Google follow these rules when deciding whether to read your site for training data or real-time answers. If you block them, they can't learn about your business, so they can't recommend you.
What's the difference between "Blocked" and "Blocked (wildcard)"?: Blocked means there's a specific rule for that bot (e.g. User-agent: GPTBot / Disallow: /). Blocked (wildcard) means the bot isn't mentioned explicitly, but a catch-all rule (User-agent: *) blocks everything. Both have the same effect: the crawler won't access your site.
What does "Not specified" mean?: No explicit rule exists for that bot, either in its own block or via a wildcard Disallow: /. In practice this means the crawler is allowed to access your site, but you haven't explicitly told it so. Most AI bots will crawl in this case, though being explicit is better practice.
Should I allow all AI bots?: For most businesses, yes. This matters most for bots that power search and recommendations (GPTBot, Perplexitybot, anthropic-ai). Blocking them means AI engines can't read your site, which makes it harder for them to represent you accurately in responses. The exception is if you have a specific reason to restrict training data use, in which case you can block training-focused bots while still allowing search and browsing bots.
Do AI companies actually respect robots.txt?: Major AI companies, including OpenAI, Anthropic, Google, and Perplexity, publicly commit to respecting robots.txt. Some smaller or less reputable crawlers may not. For the bots listed in this tool, compliance is generally expected. Still, robots.txt is advisory. It works only when the crawler chooses to comply.
How do I update my robots.txt to allow AI bots?: Add explicit allow rules for the bots you want to grant access. For example, to allow OpenAI's crawlers add User-agent: GPTBot followed by Allow: / on the next line. If you have a wildcard Disallow: /, add the specific bot rules before the wildcard block. More specific rules take precedence.