Three crawlers, three jobs
Claude-SearchBot crawls to improve search results for Claude users; Anthropic says blocking it stops your content being indexed for search, which may reduce your site's visibility and accuracy in user search results.[1]
Claude-User fetches pages when a person using Claude asks a question; blocking it stops your content being retrieved for those queries, which may reduce your visibility for user-directed web search.[1]
ClaudeBot collects content that could contribute to training Anthropic's models. Blocking it signals that your future content should be left out of training data.[1]
How sites are included or left out
Anthropic says its bots honour robots.txt, set per subdomain, and warns that blocking its IP addresses may not reliably work as an opt-out.[1]
A noindex robots meta tag tells Anthropic's search partners not to index a page, so it won't appear in Claude answers that use web search. Anthropic doesn't name those partners.[2]
If your robots.txt or a security setting blocks Claude-SearchBot or Claude-User, Claude has a harder time finding or reading your site, however good the page is.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Claude Help Center)
- Report, block, and remove content from Claude (Claude Help Center)
Numbered points come from these pages, read on 23 October 2026. Lines marked “Our reading” are our interpretation, not the company's words. CiteArtemis isn't affiliated with Anthropic.