GPTBot
OpenAI's web crawler, which gathers content used to train and inform its models; site owners can allow or block it via robots.txt.
GPTBot is the crawler OpenAI uses to collect web content. Site owners can control its access through robots.txt, allowing or disallowing it like any other user agent. Other AI crawlers (such as Common Crawl’s CCBot) work similarly.
The trade-off
Blocking AI crawlers protects content from being ingested, but it can also reduce the chance that your information is available for an AI system to represent and cite. For most businesses that want to be recommended accurately, allowing access is usually the goal.
How to manage it
- Decide deliberately whether to allow or block each AI crawler
- Use
robots.txtto set per-user-agent rules - Revisit the policy as AI traffic and citation behavior evolve
Practical takeaway
For an AEO strategy, blocking AI crawlers is usually counterproductive — you generally want models to have accurate access to your content.
Related Terms
- Optimization Tactics
Content Chunking
Structuring content into self-contained, clearly-labeled sections so AI systems can extract and cite a specific passage without needing the whole page.
- Optimization Tactics
llms.txt
A proposed standard file placed at a site's root that offers AI systems a curated, plain-text map of the site's most important content.
- Optimization Tactics
Schema Markup
Structured data added to a web page using a shared vocabulary (schema.org) that explicitly tells search engines and AI systems what the content means.