Part of How to Get Your Website Into ChatGPT Search
ClaudeBot, Claude-SearchBot and Claude-User: Anthropic's Crawlers
ClaudeBot collects training data; Claude-SearchBot and Claude-User serve Claude's search and page fetches. What each does and robots.txt rules for each.
ClaudeBot is Anthropic's web crawler: it collects publicly available pages that may be used to train Claude models. It's one of three user agents Anthropic documents. Claude-SearchBot crawls to improve Claude's search results, and Claude-User fetches pages when someone asks Claude a question. Each one reads its own robots.txt group, so you can opt out of training without dropping out of Claude's answers.
This page is about Anthropic's crawlers, not Clawdbot, the open-source AI assistant now called OpenClaw. OpenAI splits its bots the same way (OAI-SearchBot, GPTBot and ChatGPT-User), and the wider picture of AI search starts with how to get your website on ChatGPT.
ClaudeBot, Claude-SearchBot and Claude-User at a glance
Anthropic's help page Does Anthropic crawl data from the web, and how can site owners block the crawler? lists the three agents with what each does and what blocking it does. As of October 2026:
| Agent | What it does | What blocking it does |
|---|---|---|
ClaudeBot | Collects public web content that may be used to train Anthropic's models | Signals that your future content should be excluded from training datasets |
Claude-SearchBot | Crawls the web to improve the quality of Claude's search results | Stops your content being indexed for search; may reduce your visibility and accuracy in Claude's search results |
Claude-User | Fetches a page when a person's question to Claude needs it | Stops Claude retrieving your content for a user's query; may reduce your visibility in user-directed search |
The same page says Anthropic's bots honor robots.txt, support the non-standard Crawl-delay extension and won't try to bypass CAPTCHAs. It names only these three tokens. Some older blocklists carry anthropic-ai and Claude-Web; Anthropic's page mentions neither, so a robots.txt that names only those doesn't cover the agents Anthropic documents today.
ClaudeBot: training data
ClaudeBot is about training, not search. Blocking it is, in Anthropic's words, a signal that your future materials should be excluded from its training datasets. The page doesn't say anything about content collected before you added the rule, and it doesn't tie ClaudeBot to search visibility: that's the job of the other two.
Claude-SearchBot: search
Claude-SearchBot crawls to improve search result quality. The help page doesn't say how often it crawls or how what it finds is ranked. Its warning is the useful part: block it and your site may lose visibility and accuracy in Claude's search results.
Claude-User: fetches on a user's behalf
Claude-User fetches a page because a person asked Claude something that needs it. Claude Code uses it too: its changelog for version 2.1.83 (March 25, 2026) says its WebFetch tool identifies as Claude-User, so site owners can recognize Claude Code requests and allow them in robots.txt. Blocking Claude-User means Claude can't read your page even when a user points it straight at it.
How Claude finds websites
Claude doesn't search for every question. Anthropic's web search tool docs say Claude decides when to search based on the prompt, typically for recent events, current prices, and organizations or products that may have changed, and answers stable knowledge directly. The search runs, Claude reads the results, and the final answer cites its sources. In the Claude apps, the web search help article says every response that uses web search includes citations.
The same help article says image results are powered by Bing; it doesn't name the provider behind text results, and Anthropic's crawler page doesn't either. In March 2025, TechCrunch reported that Anthropic had added Brave Search to its subprocessor list the week web search launched in Claude; neither company confirmed the link in that report. If Brave is part of it, one detail matters: Brave's crawler help page says its bot has no distinct user agent and won't crawl a domain or page that Googlebot can't crawl. A robots.txt rule that keeps Googlebot out keeps Brave out too. Brave Search indexing covers that index.
Reading a specific page is a fetch. For the API's web fetch tool, Anthropic documents that Claude can only fetch URLs that already appeared in the conversation (a user message, a tool result or an earlier search), that the tool doesn't support sites rendered with JavaScript, that results are cached and may not be the latest version, and that a URL can be refused because of robots.txt. Anthropic doesn't say how Claude-SearchBot handles JavaScript, so server-rendered HTML is the safe choice (more in JavaScript SEO).
Put together, a page Claude can cite needs three things:
- It's in the search index Claude's search draws on, which starts with being crawlable at all.
- Claude-SearchBot and Claude-User can reach it: robots.txt, CDN and firewall all let them through.
- Its main content is in the HTML the server returns, not only in what JavaScript builds later.
Robots.txt rules for Anthropic's crawlers
Robots.txt matches on the product token, case-insensitively, and a crawler follows the group that names it. A crawler with no group of its own falls back to User-agent: * (RFC 9309). Anthropic asks for the rules in the robots.txt of every subdomain you want to opt out. The general mechanics are in robots.txt for SEO.
Opt out of training, stay in Claude's answers
User-agent: ClaudeBot
Disallow: /
User-agent: *
Disallow: /admin/ClaudeBot is out. Claude-SearchBot and Claude-User have no group, so they follow the * group like every other well-behaved crawler.
Watch the fallback when you add a named group
A crawler that matches a named group ignores the * group entirely. Google's robots.txt specification says the same for its own crawlers: one group applies, and the global group isn't merged in. So this opens /admin/ to Claude-SearchBot:
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Disallow: /admin/If you give an Anthropic agent its own group, repeat in it the Disallow lines you still want it to obey. The same applies to a group that only sets a crawl delay.
Block all three
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /One group can list several user agents. This keeps your pages out of Anthropic's training data and, by Anthropic's own account, makes them less visible in Claude's search results and unreadable when a user asks Claude about them.
Slow ClaudeBot down instead
User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /admin/Anthropic supports Crawl-delay and gives Crawl-delay: 1 as its example; the Disallow line is there because of the fallback rule above. Google doesn't: its specification says crawl-delay isn't supported.
Serve robots.txt with a 200
Under RFC 9309, a robots.txt that answers 4xx lets a crawler access anything, and one that answers 5xx means it must assume the whole site is disallowed. Anthropic adds that blocking its IP addresses may not keep you opted out, because the block also stops its bots reading your robots.txt. Use robots.txt for the policy. For a wider policy across AI vendors, see which AI crawlers to allow in robots.txt.
Check your CDN and firewall, not just robots.txt
Anthropic says its bots respect anti-circumvention measures and won't bypass CAPTCHAs, so a bot challenge stops them as surely as a block. On Cloudflare, as of October 2026, AI bots are handled under Security Settings > Configure AI bot policies, with separate Search, Agent and Training settings, each of which can block on all pages, block only on pages with ads, or allow. Cloudflare's AI bots docs say new domains get defaults from September 15, 2026 that block Training and Agent bots on pages with ads and allow Search. The page doesn't say which category each Anthropic agent falls under, so check how your settings treat all three.
Cloudflare can also edit robots.txt for you. With Bot Preference Sync on and a policy set to block, its managed robots.txt prepends Disallow: / groups for AI crawlers above your own rules. ClaudeBot is in Cloudflare's example of that block; the page doesn't say whether Claude-SearchBot or Claude-User get added. If you find an Anthropic rule you never wrote, that's one likely source. Firewall rules that catch AI bots can catch Googlebot too: Blocked due to access forbidden (403) covers that case.
How to see whether Anthropic's crawlers visit your site
Search your access logs for the three tokens, then check where the requests came from. Anthropic publishes the IP prefixes its crawlers use at claude.com/crawling/bots.json, a JSON list of ipv4Prefix entries:
# Requests per Anthropic agent
grep -oE "ClaudeBot|Claude-SearchBot|Claude-User" access.log | sort | uniq -c
# Source IPs of requests claiming to be ClaudeBot (combined log format)
grep "ClaudeBot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head
# Anthropic's published prefixes
curl -s https://claude.com/crawling/bots.json | jq -r '.prefixes[].ipv4Prefix'A user agent string is easy to fake, so the IP is the evidence. Anthropic's help page states the positive case: a request from an IP on the list comes from Anthropic. If your logs show Claude-SearchBot or Claude-User getting 403 or challenge responses, they aren't getting those pages, whatever your robots.txt says.
Where IndexCyborg fits
IndexCyborg doesn't monitor Claude or Anthropic's crawlers: it doesn't read your server logs, and its robots.txt check evaluates your rules for Googlebot. It works on the search-engine side instead: it discovers your pages from sitemaps, notifies the IndexNow engines when pages are new or changed (once you've verified the site and its IndexNow key), inspects each page's Google status with the URL Inspection API and flags pages blocked by robots.txt or noindex. Given Brave's rule above, a page IndexCyborg reports as blocked for Googlebot is one Brave's crawler skips too.
If you build with Claude Code, you can ask from the terminal. Connect the MCP server with claude mcp add --transport http indexcyborg https://indexcyborg.com/api/mcp --header "Authorization: Bearer ic_your_key" and ask which pages Google hasn't indexed and why; Claude Code MCP walks through the setup. More on the features page.
Frequently asked questions
Does ClaudeBot respect robots.txt?
Yes, according to Anthropic. Its help page says Anthropic's bots honor robots.txt directives, support the non-standard Crawl-delay extension and don't try to bypass CAPTCHAs. Rules are read per host, so each subdomain needs its own robots.txt.
Does blocking ClaudeBot remove my site from Claude?
No. Anthropic says blocking ClaudeBot signals that your future content should be excluded from training. It says nothing about content already collected, and the rule doesn't apply to Claude-SearchBot or Claude-User, which serve Claude's search and page fetches.
Does blocking ClaudeBot affect Google rankings?
No. Google's crawlers follow only the robots.txt group that matches them most specifically, so a ClaudeBot group doesn't change what Googlebot may crawl. Keep the rule in its own ClaudeBot group, not under User-agent: *.
What is the ClaudeBot user agent?
Anthropic documents the tokens ClaudeBot, Claude-SearchBot and Claude-User rather than full user agent strings, and the token is what robots.txt matches on. Anyone can send those tokens, so check the source IP against Anthropic's published list at claude.com/crawling/bots.json.
Why is ClaudeBot crawling my site so much?
ClaudeBot collects public pages for model training, so it can request many URLs on a large site. Anthropic supports Crawl-delay in robots.txt to slow it down, or you can disallow it. Anthropic warns that blocking its IP addresses can stop it from reading robots.txt, so the opt-out may not hold.