Your Analytics Can't See AI Crawlers. Cloudflare Can: A Guide to beket.ai's Cloudflare Integration

AI crawlers do not run JavaScript.

Google Analytics runs on JavaScript. So GA never sees GPTBot, ClaudeBot or PerplexityBot. It sees the humans who click through from an AI answer. It misses every bot visit before that click.

Cloudflare sits in front of your site. Every request passes through it, bot or human. The log records which bot asked for which page.

beket.ai reads that log. The integration is free on every plan.

What you get

The integration adds two data sets to your audit:

AI-crawler traffic. Which AI bots reached your site, how often, and which pages they read.
AI-referral traffic. Humans who arrived from an AI answer, such as ChatGPT.

You need a site on Cloudflare. That is the only requirement.

Connect Cloudflare in three steps

Open your beket.ai settings. Find the Cloudflare card.

1. Create an API token. In Cloudflare, open My Profile → API Tokens. Click Create Token. The beket.ai card links straight to that page.

2. Set three read permissions. Scope the token to the site you want measured. Give it:

Zone → Analytics → Read
Zone → Zone → Read
Account → Account Analytics → Read

All three are read-only. beket.ai cannot change your Cloudflare settings.

3. Paste the token and pick the zone. Paste the token into the card. Click Find zones. Pick your site. Click Connect.

The card now shows your domain, a Connected badge, the last update date, and how many audits use it.

The AI Activity summary

Your audit dashboard gets an AI Activity panel. It has two cards.

AI crawler requests. Our own site logged 929 requests over 7 measured days, from 10 AI crawlers. The card splits them by why the bot came:

Training: 551. A bot collecting text to train a model.
Indexing: 305. A bot building a search index. Indexing decides whether you can be cited.
User intent: 73. Someone asked an AI a question. The AI went to read your page right then.

Below the split are the top three crawlers. For us: ClaudeBot 321, OAI-SearchBot 246, Amazonbot 179.

Visits from AI answers. Humans who clicked through from an AI tool. We had 12 sessions over 35 days.

Click View all for the full report.

The full report: AI Crawl Traffic

The AI funnel

The funnel has three columns.

The left column counts crawler requests at the Cloudflare edge. Over 30 days we saw 7,749 requests. ClaudeBot led with 1,701. Amazonbot sent 1,005. OAI-SearchBot sent 980. GPTBot sent 689. Twelve more bots sent 3,374.

The right column counts human arrivals by AI surface. Google Analytics logged 12 ChatGPT sessions. Cloudflare logged 1 ChatGPT pageview.

The middle column is blank on purpose. No tool measures whether a fetch became a citation. No tool measures whether a citation became a click. We do not guess at it.

The two arrival counts stay apart. A session and a pageview count different things. One visit can be both. Adding them would count it twice.

The daily chart

Bars show crawler requests per day. Each bar stacks three kinds: Live, Search and Training. Lines show human arrivals.

The chart uses two axes. Bot requests sit on the left axis. Human arrivals sit on the right. The two are not comparable, so they do not share a scale.

A day with no bar was never read. It does not mean zero.

Coverage

The coverage strip shows one mark per day, newest first:

✓ measured, with hits
· measured, nothing found
✗ not measured

Cloudflare’s free plan keeps 8 days of logs. beket.ai only asks for the days it is missing. Run the integration on a free plan and older days fill in over time. A row of ✗ at the old end of the strip is expected. It is not a fault.

Crawlers

The crawler table lists every AI bot that reached you. Each row shows:

Kind. Training, Search or Live.
Requests. Total, plus how many Cloudflare blocked.
Days seen. How many days the bot showed up.
Sampling. Cloudflare samples some traffic. 1:2 means one logged request stands for two.
Detector. How the bot was named. Today that is the user-agent string.

The Live rows matter most. ChatGPT-User, Perplexity-User and Claude-User fetch a page because a person asked a question. On our site ChatGPT-User made 575 requests. Perplexity-User made 371. Claude-User made 31.

Check the blocked column. Bytespider had 16 requests blocked on our site. OAI-SearchBot had 7. A blocked search bot cannot index you. A page it cannot index cannot be cited.

Pages people are asking about right now

This card lists the pages that Live bots fetched in the last 7 days.

On our site, 73 of 929 requests were live fetches. Our home page took 23 live fetches out of 119 requests. Our /mcp page took 19 live fetches out of 19. Every request to /mcp came from a person asking an AI about it.

The log records the bot and the page. It never records the question.

Pages reached

This table crosses pages with crawlers. Each cell shows how many times a bot fetched a page. A dot means a measured zero.

Read it for two things.

First, find which bots read your money pages. On our site ClaudeBot read the home page 305 times. ChatGPT-User read our pricing page 21 times. GPTBot read it 4 times.

Second, find the junk. Our table shows requests for /.bashrc, /media…/.env and /__/firebase/init.json. Those files do not exist on our site. Real crawlers rarely ask for them. Scanners do. The detector trusts the user-agent string, and any client can send any string. Treat those rows as probes, not AI interest.

Fonts, images and /robots.txt are left out. Without that filter the list would be mostly typefaces.

What to do with the data

Unblock the bots you want citing you. Check the blocked column against your firewall and bot rules.
Fix the pages Live bots read. Those pages answer real questions today. Make them accurate.
Watch indexing bots on new pages. If OAI-SearchBot has not read a page, ChatGPT search cannot cite it.
Block the probes. Requests for .env and .bashrc are not AI traffic.

Next step

Crawler logs show which pages AI tools read. A beket.ai audit shows what those tools then say about you. Connect Cloudflare, run an audit, and see both at beket.ai.

#AISearch #AEO #GEO #Cloudflare #AICrawlers #ChatGPT #SEO #beketai