Episode 2
Technical AI bot checklist
How Do AI Search Engines Choose Sources to Cite?
Search crawlers fetch pages for citations. Training crawlers build model memory. Blocking a training crawler does not remove you from search, and allowing one does not put you there.
0 / 16 checked. Saved in this browser.
1
Allow the search crawlers
2
Decide on the training crawlers
Allow these if you want the text in future training sets. Disallow them to limit that use. Either way, write the rule on purpose.
3
Confirm the fetch
4
Markup and optional files
JSON-LD tells a machine what the entity is. An Ahrefs comparison of pages that added it and pages that did not found no meaningful change in citation rate. Ship it for clarity. Do not ship it as a ranking trick.
Search crawlers
| Token | Used for | If it cannot fetch |
|---|---|---|
| OAI-SearchBot | ChatGPT search | No citations in ChatGPT search |
| PerplexityBot | Perplexity indexing | No retrieval for Perplexity answers |
| Perplexity-User | A fetch someone triggered | Perplexity says robots.txt is generally ignored |
| Claude-SearchBot | Claude search | No citations in Claude search |
| Claude-User | A fetch someone triggered in Claude | That fetch fails |
| Googlebot | Google Search, AI Overviews, AI Mode | No organic result and no AI Overview |
| Bingbot | Bing Search and Copilot | No Bing result and no Copilot citation |
Training crawlers
| Token | Used for | Policy |
|---|---|---|
| GPTBot | OpenAI training | Allow for future model memory, or disallow to limit training |
| ClaudeBot | Anthropic training | Claude-SearchBot is the search crawler. This one is training. |
| Google-Extended | Gemini training control | Blocking it leaves Google Search and AI Overviews unchanged |
| CCBot | Common Crawl | Disallow to limit training use, or allow if you want the corpus to include you |
robots.txt
Paste the allow block. Uncomment a training policy only after you have chosen one.
# Search crawlers. Allow these so AI search can fetch and cite the page.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Training crawlers. Pick a policy on purpose. Leaving them commented changes nothing.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: CCBot
# Allow: /
# Disallow: /Questions
- Which token gets a site into ChatGPT search?
- OAI-SearchBot. GPTBot trains models. Blocking GPTBot does not block ChatGPT search.
- Does ClaudeBot need to be allowed for Claude citations?
- Claude-SearchBot fetches pages for Claude search. ClaudeBot trains models. Allow Claude-SearchBot. Choose ClaudeBot as a training decision.