Blog · technical
AI crawlers: which bots to allow, block and why
Training crawlers and retrieval crawlers are different bots doing different jobs, and most blocking decisions treat them as one. Blocking a training crawler costs you nothing you can measure. Blocking a retrieval crawler makes citation impossible.
Reference
What each bot actually does
The user agent string is the whole decision. These are separate directives in robots.txt and can be set independently.
| User agent | Operator | Job | Blocking it costs you |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Builds the index ChatGPT search answers from | Any possibility of a ChatGPT citation |
| GPTBot | OpenAI | Collects training data | Presence in a future model's memory |
| ChatGPT-User | OpenAI | Fetches a page live during a user request | The page loading when a user asks about it |
| ClaudeBot | Anthropic | Collects training data | Presence in a future model's memory |
| Claude-SearchBot / Claude-User | Anthropic | Retrieval and live user fetches | Citation in Claude answers |
| PerplexityBot | Perplexity | Builds its retrieval index | Citation in Perplexity answers |
| Google-Extended | Controls Gemini training use only | Nothing in Search or AI Overviews | |
| Googlebot | Search index, and the basis of AI Overviews | Search and AI Overviews together |
Google-Extended is the one people get wrong in the expensive direction. It governs training use, not Search. Blocking it does not remove you from AI Overviews, because AI Overviews are built on the ordinary Googlebot crawl. There is no separate opt-out for AI Overviews that keeps you in Search.
Check this first
The default that blocks you without anyone deciding to
Before any content work is commissioned, check whether the site is reachable at all. On a large share of sites it is not, and nobody in the marketing team made that call.
Since 1 July 2025, new domains on Cloudflare block GPTBot, ClaudeBot and PerplexityBot by default, and the setting does not distinguish training crawlers from retrieval crawlers.
This is the most common cause of a site being invisible to AI answers for reasons nobody chose. Check it before commissioning any AEO work.
The setting is a single toggle labelled for training crawlers, and it takes the retrieval crawlers with it. A site can be perfectly written, perfectly structured, fully schema-marked and completely uncitable because of an infrastructure default set on the day the domain was onboarded.
- Fetch
/robots.txtand read it. Look forDisallowunder each AI user agent by name, not just under*. - Check the CDN or WAF layer separately. A permissive robots.txt means nothing if the edge returns 403 to the bot.
- Check server logs or CDN analytics for actual hits from OAI-SearchBot, PerplexityBot and Googlebot. Permission is a claim; a 200 in the log is evidence.
- Check for JavaScript-dependent content. Retrieval crawlers are not guaranteed to execute it, and a page whose text arrives only after hydration may be an empty page to them.
The data
Who is being blocked, and how much
Blocking is now common enough to be a market-shaping fact rather than an edge case.
In a snapshot of 4,223 robots.txt files taken on 27 July 2026, GPTBot was disallowed on 633 domains, CCBot on 567, ClaudeBot on 563, Google-Extended on 522 and Bytespider on 518. By declared purpose, 44.54% of AI crawler traffic was for training, 39.99% mixed, 11.57% search and 2.66% user-initiated.
Note the purpose split in that data. Training and mixed-purpose crawling together account for 84.5% of AI crawler traffic; search and user-initiated fetches account for 14.2%. Most of what hits your server has no path back to you at all — but the minority that does is exactly the traffic a blanket block removes.
AI crawlers made up 20.3% of verified bot traffic in May 2026, with AI-search bots adding a further 6.5%. GPTBot accounted for 11.48% of AI bot requests and ClaudeBot 9.73%, reversing April's order.
The exchange
How much they take for what they return
The case for blocking is usually made on this number, so it is worth quoting honestly, including where the sources disagree.
Pages crawled per referral sent back, July 2026: Anthropic 1,917:1, Perplexity 289:1, OpenAI 251:1, against Google at 4.7:1. A second tracker reading Cloudflare data reports far steeper ratios for the same period, including ClaudeBot at 11,122:1 and GPTBot at 1,276:1.
The two sources disagree by an order of magnitude and both are carried rather than one being picked. What they agree on is the shape: AI crawlers take far more than they return, and Google returns orders of magnitude more traffic per page crawled than any of them.
Source conflict, declared. The two trackers differ by roughly an order of magnitude on the same crawlers in the same period — Anthropic at 1,917:1 versus 11,122:1, OpenAI at 251:1 versus 1,276:1. They are measuring different site panels. Neither is quoted here as the figure. What survives the disagreement is the shape: every AI crawler takes vastly more than it returns, and Google returns two to three orders of magnitude more traffic per page crawled than any of them.
That is a genuine argument for blocking training crawlers, and publishers with real bandwidth costs and licensing leverage are right to make it. It is not an argument for blocking retrieval crawlers, which is the decision the common defaults actually make for you.
Deciding
A defensible default policy
Most organisations selling something, rather than licensing content, land in roughly the same place.
-
01
Allow every retrieval and user-action crawler
OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Googlebot. These are the only bots that can produce a citation or a visit. There is no upside to blocking them unless you are deliberately withholding content.
-
02
Decide training crawlers on principle, not on traffic
GPTBot, ClaudeBot, CCBot, Google-Extended. Allowing them is a bet on being in a future model's memory, which is real but unmeasurable and slow. Blocking them costs nothing you can currently observe. Either answer is defensible; pick one and write down why.
-
03
Block scrapers with no answer surface
Bytespider and similar bots that neither cite nor refer. Nothing is given up.
-
04
Verify at the edge, not just in robots.txt
Confirm the CDN, WAF and bot-management rules agree with the file. This is where the policy is usually contradicted.
-
05
Re-check quarterly
User agents get added, platform defaults change, and a security review can revert the whole thing in one commit. Crawler access is a state to be monitored, not a task to be completed.
The file that does not help
llms.txt is not part of this
It comes up in every crawler conversation, so it is worth settling.
Google documentation updated in June 2026 states llms.txt has no effect on Search rankings or AI Overviews. John Mueller noted that no AI crawler has claimed it extracts information from the file.
A study of 300,000 domains found llms.txt on 10.13% of them; a separate study of 137,000 sites found 97% of llms.txt files received zero traffic in May 2026.
No major answer engine has claimed to read it, Google has documented that it has no effect on Search or AI Overviews, and the overwhelming majority of published files receive no traffic at all. Publishing one is harmless. Counting it as AI visibility work is not, because it displaces the crawler-access check that actually decides whether you can be cited.
Sources
Where these numbers come from
Every figure on this page is external and linked. The studies measure different populations over different windows, so read them as converging evidence rather than one authoritative number.
- CloudflareSince 1 July 2025, new domains on Cloudflare block GPTBot, ClaudeBot and PerplexityBot by default, and the setting does not distinguish training crawlers from retrieval crawlers. www.digitalapplied.com ↗
- Technology Checker, July 2026In a snapshot of 4,223 robots.txt files taken on 27 July 2026, GPTBot was disallowed on 633 domains, CCBot on 567, ClaudeBot on 563, Google-Extended on 522 and Bytespider on 518. By declared purpose, 44.54% of AI crawler traffic was for training, 39.99% mixed, 11.57% search and 2.66% user-initiated. technologychecker.io ↗
- Technology Checker and Digital Applied, reading Cloudflare RadarPages crawled per referral sent back, July 2026: Anthropic 1,917:1, Perplexity 289:1, OpenAI 251:1, against Google at 4.7:1. A second tracker reading Cloudflare data reports far steeper ratios for the same period, including ClaudeBot at 11,122:1 and GPTBot at 1,276:1. technologychecker.io ↗
- Google Search RelationsGoogle documentation updated in June 2026 states llms.txt has no effect on Search rankings or AI Overviews. John Mueller noted that no AI crawler has claimed it extracts information from the file. baselinelabs.ai ↗
- SE Ranking and Ahrefs studies, reported by Digital AppliedA study of 300,000 domains found llms.txt on 10.13% of them; a separate study of 137,000 sites found 97% of llms.txt files received zero traffic in May 2026. www.digitalapplied.com ↗
Keep reading
FAQ
Common questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data used for training OpenAI models. OAI-SearchBot builds the search index that ChatGPT cites from when web search is on. They are separate user agents with separate robots.txt directives, and only blocking OAI-SearchBot removes your ability to be cited in ChatGPT answers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls whether your content is used for Gemini model training. AI Overviews are generated from the ordinary Googlebot crawl of the Search index, so the only way to leave AI Overviews is to leave Search.
Is my site blocking AI crawlers without me knowing?
Quite possibly. Since 1 July 2025, new domains on Cloudflare block GPTBot, ClaudeBot and PerplexityBot by default, and the control does not separate training crawlers from retrieval crawlers. Check robots.txt, the CDN bot rules and your server logs before assuming access.
Should I block AI crawlers?
Block training crawlers if you object to unpaid training use — the measurable cost is close to zero. Do not block retrieval crawlers unless you are deliberately withholding your content from AI answers, because they are the only ones that can cite you or send a visit.
How much traffic do AI crawlers actually send back?
Very little. Trackers reading Cloudflare data in July 2026 put pages crawled per referral at 1,917:1 for Anthropic, 289:1 for Perplexity and 251:1 for OpenAI, against 4.7:1 for Google. A second tracker reports far steeper ratios on a different panel; the disagreement is large but the direction is not in dispute.
Does llms.txt help AI crawlers find my content?
There is no evidence that it does. Google documentation updated in June 2026 states llms.txt has no effect on Search rankings or AI Overviews, no major AI crawler has claimed to extract from it, and one study of 137,000 sites found 97% of published files received zero traffic.
Do AI crawlers execute JavaScript?
Not reliably, and not all of them. Content that exists only after client-side hydration may be invisible to a retrieval crawler even when the bot is fully allowed. Server-render the text you want quoted, or check what the crawler actually receives.
Not sure which bots your site is turning away? <span class='tone'>Lifewood Data Technology will audit it.</span>
Crawler access is the cheapest and most commonly broken part of AI visibility, and it is usually broken by a default nobody chose.
- 100+ languages
- 50+ delivery centres
- Measured monthly
Lifewood Data Technology publishes this reference.