# Technical AI bot checklist

Episode 2. How Do AI Search Engines Choose Sources to Cite?

Search crawlers fetch pages for citations. Training crawlers build model memory. Blocking a training crawler does not remove you from search, and allowing one does not put you there.

Playbook: [the AI search playbook](https://beket.ai/guides/ai-search-optimization-playbook/)

## 1. Allow the search crawlers

- [ ] OAI-SearchBot is allowed. Blocking it keeps the site out of ChatGPT search.
- [ ] PerplexityBot is allowed, and the firewall allows Perplexity's published IP ranges.
- [ ] Perplexity-User is allowed. Perplexity says this fetcher generally ignores robots.txt, because a person triggered the request.
- [ ] Claude-SearchBot and Claude-User are allowed.

  Claude-SearchBot is the crawler for Claude search. ClaudeBot, further down, is the training crawler.

- [ ] Googlebot is allowed. AI Overviews and AI Mode need it.
- [ ] Bingbot is allowed. Copilot answers draw on the Bing index.

## 2. Decide on the training crawlers

Allow these if you want the text in future training sets. Disallow them to limit that use. Either way, write the rule on purpose.

- [ ] GPTBot has an explicit allow or disallow. It trains OpenAI models.
- [ ] ClaudeBot has an explicit allow or disallow. It trains Anthropic models.
- [ ] Google-Extended has an explicit allow or disallow. Google says blocking it leaves Search and AI Overviews unchanged.
- [ ] CCBot has an explicit allow or disallow. Common Crawl is a training corpus.

## 3. Confirm the fetch

- [ ] The WAF or bot-fight rule does not challenge the search crawlers above.
- [ ] The robots.txt checker shows those search bots as allowed.

  [Open the robots.txt checker](https://beket.ai/robots-check/)

- [ ] View source on a sample page shows the answer text in the HTML.

  [Run the AI readiness check](https://beket.ai/ai-check/)


## 4. Markup and optional files

JSON-LD tells a machine what the entity is. An Ahrefs comparison of pages that added it and pages that did not found no meaningful change in citation rate. Ship it for clarity. Do not ship it as a ranking trick.

- [ ] The page has one JSON-LD graph. FAQ and article markup match the text a person can see.
- [ ] Organization markup names the brand. Article dates match the visible byline.
- [ ] If you publish /llms.txt, it is a short index for agents. Google Search does not use that file.

## Search crawlers

| Token | Used for | If it cannot fetch |
| --- | --- | --- |
| OAI-SearchBot | ChatGPT search | No citations in ChatGPT search |
| PerplexityBot | Perplexity indexing | No retrieval for Perplexity answers |
| Perplexity-User | A fetch someone triggered | Perplexity says robots.txt is generally ignored |
| Claude-SearchBot | Claude search | No citations in Claude search |
| Claude-User | A fetch someone triggered in Claude | That fetch fails |
| Googlebot | Google Search, AI Overviews, AI Mode | No organic result and no AI Overview |
| Bingbot | Bing Search and Copilot | No Bing result and no Copilot citation |

## Training crawlers

| Token | Used for | Policy |
| --- | --- | --- |
| GPTBot | OpenAI training | Allow for future model memory, or disallow to limit training |
| ClaudeBot | Anthropic training | Claude-SearchBot is the search crawler. This one is training. |
| Google-Extended | Gemini training control | Blocking it leaves Google Search and AI Overviews unchanged |
| CCBot | Common Crawl | Disallow to limit training use, or allow if you want the corpus to include you |

## robots.txt

Paste the allow block. Uncomment a training policy only after you have chosen one.

```
# Search crawlers. Allow these so AI search can fetch and cite the page.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Training crawlers. Pick a policy on purpose. Leaving them commented changes nothing.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: CCBot
# Allow: /
# Disallow: /
```

## Rest of the series

- Episode 1: [GEO audit checklist](https://beket.ai/resources/geo-audit-checklist/)
- Episode 3: [GEODATA audit template](https://beket.ai/resources/geodata-audit/)
- Episode 4: [AI search ROI template](https://beket.ai/resources/ai-search-roi/)