I added IndexNow to this site and my affiliate sites a while back, the setup is in IndexNow vs Google Search Console. That project got me looking at server logs more carefully, and the thing I actually noticed wasn't Googlebot. It was a growing list of other user agents I hadn't paid attention to before: GPTBot, ClaudeBot, PerplexityBot, a few others. Six months later I've got a clear enough picture of how each one behaves to write it down.
This isn't a traffic report with exact numbers, because I don't have a clean, verified external dataset to hand you, and I'd rather tell you what I actually watched than dress up a guess as data. It's what the crawl pattern looks like, which bots I allow and why, and the two or three things I'd check before anyone blocks one of these out of caution.
The bots that actually show up
Three companies run the crawlers worth knowing by name right now. Each one splits its crawling into at least two jobs: a bot that ingests content for model training, and a separate bot that fetches a page live because a user asked a question that needs it. That split matters more than the bot names themselves, because the two jobs call for different robots.txt decisions.
| User agent | Company | What it's doing |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train future models |
| OAI-SearchBot | OpenAI | Indexes pages that can be surfaced in ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page live because a user's prompt pointed at it |
| ClaudeBot | Anthropic | Crawls content for model training |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search feature |
| Claude-User | Anthropic | Fetches a page live on a user's behalf |
| PerplexityBot | Perplexity | Crawls and indexes for Perplexity's search results |
| Perplexity-User | Perplexity | Live fetch triggered by a user's question |
| Google-Extended | A permission token, not a separate crawler, it governs Gemini training use specifically, separate from Googlebot search indexing |
The "-User" and "-SearchBot" variants showing up at all is relatively recent. A lot of robots.txt templates floating around still only list GPTBot and ClaudeBot and miss the rest, which means plenty of sites are unknowingly blocking some of these and allowing others with no real logic behind the split.
What the six-month pattern actually looked like
I'm not going to hand you a chart with invented percentages. What I can tell you honestly: crawl frequency from these bots went from "a handful of hits a week, easy to miss" to "a steady, daily presence in the logs" over the six months I watched. ClaudeBot and GPTBot were the two I saw most consistently. PerplexityBot showed up in shorter, bursty windows rather than a steady daily drip, which tracks with it being tied more to live search queries than a slow training crawl.
The clearest shift wasn't volume, it was which pages got hit. Early on, these bots crawled almost everything, service pages, old blog posts, all of it. Over time the hits concentrated on pages with clean structure already in place, answer-first intros, FAQPage schema, tables instead of walls of prose. That lines up with what I found auditing actual citations in Google AI traffic is real, structure is what gets a page picked up and reused, not just existing.
I also noticed the gap between a crawl and a referral. These bots hit a page far more often than they ever send a human back to it. A training crawl doesn't send anyone anywhere, by definition, and even the search-oriented bots mostly answer the question inline without a click. That's not a complaint, it's just the trade you're making. You're optimizing for being the source an answer is built from, not for the click-through numbers classic SEO trained me to expect. If you're judging this channel by referral sessions alone, you'll conclude it's not working months before it actually isn't.
Blocking a training bot doesn't touch the search bot that sends you citations and occasional referral clicks. I've seen sites block "GPTBot" in robots.txt thinking they've opted out of ChatGPT entirely, and then still show up in ChatGPT search results because OAI-SearchBot was never named in the block. If your goal is "keep my content out of training sets," that's a different robots.txt rule than "keep my content out of AI search entirely." Decide which one you actually want before you write the rule.
My robots.txt, and why it looks the way it does
I allow all of them right now, training bots included. My content is already public and already indexed by Google, so the marginal downside of a training crawl is low for me, and the upside, showing up in ChatGPT, Claude, and Perplexity answers, is traffic I didn't have access to two years ago. This is a judgment call specific to a site like mine, not a universal recommendation.
# AI crawlers, allowed, content is public
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
User-agent: Applebot-Extended
Allow: /
# Block spam bots
User-agent: MJ12bot
Disallow: /
User-agent: DotBot
Disallow: /
The two entries most people skip are anthropic-ai and Applebot-Extended.
Neither shows up in every copy-pasted robots.txt template, and both are worth including explicitly
rather than assuming a wildcard rule covers them. I keep the spam bots, MJ12bot and DotBot, blocked
because they're link-scraping tools with no upside for a site like mine, not because they're AI related.
What I'd check before blocking any of these
None of this is a reason to block or allow anything blindly. Before you touch robots.txt on any site you run, work through these in order. It takes less time than writing the rule itself.
- Separate training from search in your head first. Decide whether your actual concern is model training or AI search visibility, they need different rules, and conflating them is the most common mistake I see in other people's robots.txt files.
- Check what you're already blocking by accident. A lot of default robots.txt files from page builders or old CMS templates block everything non-Google by default. If you've never touched the file, open it and read it, don't assume it's permissive.
- Watch your logs for a few weeks before deciding anything. You can't make a good call on volume or value from a single day's snapshot. I waited months, not days, before forming an opinion on which bots were worth allowing.
- Remember robots.txt is a request, not a lock. It's a politeness convention. A "-User" agent fetching a page because someone pasted your URL into a chat window will often ignore a disallow rule entirely, because the fetch is user-initiated, not an automated crawl. Don't treat robots.txt as access control for anything sensitive.
Key takeaways
- The bot names split into training crawlers (GPTBot, ClaudeBot, PerplexityBot) and live-fetch or search variants (ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, Claude-SearchBot). Blocking one doesn't block the other.
- Crawl frequency from these bots went from occasional to a steady daily presence over six months, PerplexityBot stayed burstier than the other two.
- Over time, crawl attention concentrated on pages with clean structure, answer-first intros, schema, tables, not just on pages that existed longest.
- My own robots.txt allows all the major AI crawlers, including training bots, because my content is already public and the AI search upside outweighs the training downside for a site like mine.
- Robots.txt is a convention, not enforcement. Treat anything you actually need to keep private as a problem for authentication, not a disallow rule.
What I'll do next
I'm going to start logging which specific pages get hit by which bot, broken out by site, instead of eyeballing the raw access logs the way I have been. That should tell me, with actual numbers this time, whether the structure-over-age pattern I'm describing qualitatively here holds up once I can measure it properly. If it does, I'll publish the real data.
FAQ
Should I block GPTBot and ClaudeBot to keep my content out of AI training? +
Does blocking these bots actually hurt my traffic?
Why do some of these bots ignore my robots.txt rules?
How do I tell these bots apart from spam scrapers in my logs?
Do I need a separate llms.txt file on top of robots.txt?
Want this built for your team?
I design AI agents and growth automation that run without babysitting. If that sounds useful, let's talk.
Get in touch โ