CCBot
User-Agent token: CCBot · Operator: Common Crawl Foundation · Category: Provides open crawl dataset, used for many purposes, includi
FraudMeNot Verdict
Listed AI Agent
CCBot builds Common Crawl, a free public web archive going back to 2008. Its data is widely used in research and in training AI models.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
Match the source IP against Common Crawl's ccbot.json ranges, and confirm IPv4 with a forward-confirmed reverse DNS lookup.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Yes.
Judgment Call · What You Should Do
Generally Low Risk on Public Pages — Verify Before Trusting
For everyday users: CCBot is a nonprofit archive crawler. Crawlers like this don't message people. Treat any message using the Common Crawl or CCBot name to demand money or logins as a scam.
For website owners & security teams: Decide whether you want your pages in a free public archive that AI developers widely reuse. If not, add the CCBot disallow from Common Crawl's page. Because fake CCBots exist, check a logged IP with reverse DNS before trusting it.
Report & Activity Summary
5
Published IP Ranges (CIDRs)
28 IPv4 addresses · operator JSON feed
2
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Yes
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
CCBot- FraudMeNot_Verdict
- Listed AI Agent
- Operator_Entity
- Common Crawl Foundation
- Agent_Category
- Provides open crawl dataset, used for many purposes, includi
- User_Agent_Spoofable
true- Verification_Method
- IP CIDR match against
https://index.commoncrawl.org/ccbot.json· Web Bot Auth: Not documented - Respects_Robots_Txt
- Yes (per ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- ALLOW or DISALLOW per your AI and archiving policy; VERIFY claimed CCBot IPs by reverse DNS; BLOCK fakes.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/ccbot/
What We Know About CCBot
CCBot is the crawler of the Common Crawl Foundation, a nonprofit that publishes a free, open copy of large parts of the web. The ai.robots.txt registry notes the archive goes back to 2008 and is cited in thousands of research papers a year. The registry also says the data is used for many purposes, including machine learning and AI, and that crawls currently run monthly.
Common Crawl's CCBot page gives its User-Agent as "CCBot/2.0 (https://commoncrawl.org/faq/)" and the opt-out as "User-agent: CCBot" with "Disallow: /". It also warns that other crawlers falsely identify themselves as CCBot, and recommends checking that the User-Agent is real.
To help with that, Common Crawl says CCBot now runs on dedicated IP address ranges with reverse DNS, except over IPv6, where reverse DNS is not yet supported. It also publishes the IPv4 and IPv6 ranges as a JSON file at index.commoncrawl.org/ccbot.json, and its file notes that forward-confirmed reverse DNS is also recommended for IPv4. You can look up a logged address and see whether it resolves to Common Crawl.
Our take: CCBot matters more than its size suggests, because Common Crawl data is free and is widely available for reuse, including for AI training. Blocking CCBot keeps new pages out of future archives. It does not remove copies already published in past crawls.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | CCBot |
| Operator | Common Crawl Foundation · Official bot documentation |
| Official IP verification feed | https://index.commoncrawl.org/ccbot.json — 5 CIDR blocks (28 IPv4 addresses), file dated 2026-08-11 |
| Sample published CIDR blocks | 2600:1f28:365:8000::/56, 3.41.188.32/29, 18.97.9.168/29, 18.97.14.80/29, 18.97.14.88/30 |
| Reverse DNS (rDNS) | Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS (not yet over IPv6), so a logged IP can be looked up to confirm it. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | Common Crawl documents a robots.txt opt-out ("User-agent: CCBot / Disallow: /"). The registry lists CCBot as respecting robots.txt. |
Copy-Paste robots.txt & Cloudflare WAF Rules for CCBot
robots.txt — allow public pages, keep private routes off-limits
User-agent: CCBot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: CCBot
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "CCBot")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Indirectly: Common Crawl publishes an open dataset that the registry says is used for machine learning and AI, among other uses |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen CCBot? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About CCBot
What is CCBot?
The crawler of the Common Crawl Foundation, which publishes a free open web archive going back to 2008.
How do I block CCBot?
Common Crawl says to add "User-agent: CCBot" and "Disallow: /" to robots.txt.
How can I tell if CCBot is real?
Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS (except IPv6) and publishes them as JSON at index.commoncrawl.org/ccbot.json. It warns that some crawlers falsely claim to be CCBot.
Could This Be a Scam? What to Do
- Anyone can copy the CCBot User-Agent. Common Crawl Foundation publishes an IP list for it, so compare the visiting address against that list before you trust the visit.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the CCBot or Common Crawl Foundation name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More AI Agents
- AddSearchBot — AI Search Crawlers
- adidxbot — Advertising quality crawler
- AgentDataBot — AI Data Providers
- AgentTimes — Data Scraper from RSS Feeds
- AGI Agent — AI Agents
- AhrefsBot — SEO crawler
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Common Crawl Foundation IP feed, Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Common Crawl: CCBot, Common Crawl: ccbot.json IP ranges. FraudMeNot is not affiliated with Common Crawl Foundation. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.