anthropic-ai: what it is and how to block or allow it
User-Agent token: anthropic-ai · Operator: Anthropic · Category: Scrapes data to train Anthropic's AI products
FraudMeNot Verdict
AI Data Collection Bot
anthropic-ai is an older Anthropic robots.txt token. Anthropic's current crawler page names only ClaudeBot, Claude-User and Claude-SearchBot, so block those tokens too.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
Anthropic publishes crawler source IPs at claude.com/crawling/bots.json. A request from outside that list should be treated as unverified (likely not from Anthropic).
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Unclear.
Judgment Call · What You Should Do
Your Choice — Your Content May Be Collected for AI
For everyday users: Crawlers don't send messages. Treat a message using the anthropic-ai or "Claude" name that asks for a password, code or payment as a likely scam.
For website owners & security teams: To opt out of Anthropic training, disallow ClaudeBot, which Anthropic documents, and keep anthropic-ai if it is already there. Verify requests against bots.json. Anthropic says IP blocks can stop it reading your robots.txt, so use robots.txt for the opt-out.
Report & Activity Summary
14
Published IP Ranges (CIDRs)
1,037 IPv4 addresses · operator JSON feed
1
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Unclear
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
anthropic-ai- FraudMeNot_Verdict
- AI Data Collection Bot
- Operator_Entity
- Anthropic
- Agent_Category
- Scrapes data to train Anthropic's AI products
- User_Agent_Spoofable
true- Verification_Method
- IP CIDR match against
https://claude.com/crawling/bots.json· Web Bot Auth: Not documented - Respects_Robots_Txt
- Unclear (not stated by the operator or the ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- DISALLOW ClaudeBot (plus anthropic-ai) to opt out of Anthropic training; verify any Anthropic-named traffic against bots.json.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/anthropic-ai/
What We Know About anthropic-ai
The anthropic-ai token still appears in many robots.txt files and in the ai.robots.txt registry, which lists it as collecting data to train Anthropic's AI products. Anthropic's current help-center page on crawling, though, does not mention anthropic-ai at all. It names three robots: ClaudeBot, Claude-User and Claude-SearchBot.
Anthropic describes them this way. ClaudeBot collects web content that could contribute to model training, and restricting it signals that a site's future materials should be left out of training datasets. Claude-User fetches pages when a person asks Claude a question. Claude-SearchBot indexes content to improve search results. Each one can be disallowed by name.
The practical point: a robots.txt that only says "User-agent: anthropic-ai" does not name any of the robots Anthropic documents today. Keep the line if you like, but add ClaudeBot (and Claude-User or Claude-SearchBot if you want) to make your opt-out explicit. See our ClaudeBot page at /bot/claudebot/ for the token Anthropic documents today.
Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay extension and will not try to bypass CAPTCHAs. It publishes a list of crawler source IPs at claude.com/crawling/bots.json. Anthropic advises against relying on IP blocks to opt out, because a block stops it reading your robots.txt.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | anthropic-ai |
| Operator | Anthropic · Official bot documentation |
| Official IP verification feed | https://claude.com/crawling/bots.json — 14 CIDR blocks (1,037 IPv4 addresses), file dated 2026-10-20 |
| Sample published CIDR blocks | 216.73.216.0/22, 34.162.230.222/32, 34.162.244.71/32, 34.162.191.81/32, 34.162.86.102/32, 136.79.143.251/32 |
| Reverse DNS (rDNS) | Not documented in the public datasets FraudMeNot uses. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | Anthropic says its bots honor robots.txt and support Crawl-delay. Its page does not name anthropic-ai; it names ClaudeBot, Claude-User and Claude-SearchBot. |
Copy-Paste robots.txt & Cloudflare WAF Rules for anthropic-ai
robots.txt — allow public pages, keep private routes off-limits
User-agent: anthropic-ai
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: anthropic-ai
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "anthropic-ai")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | The registry says so; Anthropic now names ClaudeBot as the crawler whose access controls training use |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen anthropic-ai? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About anthropic-ai
Is anthropic-ai still used?
Anthropic's current crawler page does not list it. It names ClaudeBot, Claude-User and Claude-SearchBot.
How do I block Anthropic's training crawler?
Add "User-agent: ClaudeBot" and "Disallow: /" to robots.txt for each subdomain, as Anthropic's help page shows. You can keep an anthropic-ai group as well.
Does Anthropic publish its crawler IPs?
Yes. Anthropic links to a list at claude.com/crawling/bots.json.
Does Anthropic honor Crawl-delay?
Yes. Anthropic says it supports the non-standard Crawl-delay extension, for example "User-agent: ClaudeBot" with "Crawl-delay: 1".
Could This Be a Scam? What to Do
- Anyone can copy the anthropic-ai User-Agent. Anthropic publishes an IP list for it, so compare the visiting address against that list before you trust the visit.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the anthropic-ai or Anthropic name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More from Anthropic
- ClaudeBot — Scrapes data to train Anthropic's AI products
- Claude-User — AI Assistants
- Claude-SearchBot — Claude-SearchBot navigates the web to improve search result
- Claude-Web — Undocumented AI Agents
- Claude-Code — AI Coding Agents
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Anthropic IP feed, Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Claude Help Center: Does Anthropic crawl data from the web?, Anthropic: crawler IP list (bots.json). FraudMeNot is not affiliated with Anthropic. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.