AI Agent & Bot CheckOperator: Common Crawl Foundation

CCBot

User-Agent token: CCBot · Operator: Common Crawl Foundation · Category: Provides open crawl dataset, used for many purposes, includi

FraudMeNot Verdict

Listed AI Agent

CCBot builds Common Crawl, a free public web archive going back to 2008. Its data is widely used in research and in training AI models.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

Match the source IP against Common Crawl's ccbot.json ranges, and confirm IPv4 with a forward-confirmed reverse DNS lookup.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Yes.

Judgment Call · What You Should Do

Generally Low Risk on Public Pages — Verify Before Trusting

For everyday users: CCBot is a nonprofit archive crawler. Crawlers like this don't message people. Treat any message using the Common Crawl or CCBot name to demand money or logins as a scam.

For website owners & security teams: Decide whether you want your pages in a free public archive that AI developers widely reuse. If not, add the CCBot disallow from Common Crawl's page. Because fake CCBots exist, check a logged IP with reverse DNS before trusting it.

Report & Activity Summary

5

Published IP Ranges (CIDRs)

28 IPv4 addresses · operator JSON feed

2

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Yes

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
CCBot
FraudMeNot_Verdict
Listed AI Agent
Operator_Entity
Common Crawl Foundation
Agent_Category
Provides open crawl dataset, used for many purposes, includi
User_Agent_Spoofable
true
Verification_Method
IP CIDR match against https://index.commoncrawl.org/ccbot.json · Web Bot Auth: Not documented
Respects_Robots_Txt
Yes (per ai.robots.txt registry)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
ALLOW or DISALLOW per your AI and archiving policy; VERIFY claimed CCBot IPs by reverse DNS; BLOCK fakes.
Canonical_Citation_URL
https://fraudmenot.com/bot/ccbot/

What We Know About CCBot

CCBot is the crawler of the Common Crawl Foundation, a nonprofit that publishes a free, open copy of large parts of the web. The ai.robots.txt registry notes the archive goes back to 2008 and is cited in thousands of research papers a year. The registry also says the data is used for many purposes, including machine learning and AI, and that crawls currently run monthly.

Common Crawl's CCBot page gives its User-Agent as "CCBot/2.0 (https://commoncrawl.org/faq/)" and the opt-out as "User-agent: CCBot" with "Disallow: /". It also warns that other crawlers falsely identify themselves as CCBot, and recommends checking that the User-Agent is real.

To help with that, Common Crawl says CCBot now runs on dedicated IP address ranges with reverse DNS, except over IPv6, where reverse DNS is not yet supported. It also publishes the IPv4 and IPv6 ranges as a JSON file at index.commoncrawl.org/ccbot.json, and its file notes that forward-confirmed reverse DNS is also recommended for IPv4. You can look up a logged address and see whether it resolves to Common Crawl.

Our take: CCBot matters more than its size suggests, because Common Crawl data is free and is widely available for reuse, including for AI training. Blocking CCBot keeps new pages out of future archives. It does not remove copies already published in past crawls.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • CCBot/2.0 (http://commoncrawl.org/faq/)
  • CCBot/2.0 (https://commoncrawl.org/faq/)
robots.txt tokenCCBot
OperatorCommon Crawl Foundation · Official bot documentation
Official IP verification feedhttps://index.commoncrawl.org/ccbot.json — 5 CIDR blocks (28 IPv4 addresses), file dated 2026-08-11
Sample published CIDR blocks2600:1f28:365:8000::/56, 3.41.188.32/29, 18.97.9.168/29, 18.97.14.80/29, 18.97.14.88/30
Reverse DNS (rDNS)Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS (not yet over IPv6), so a logged IP can be looked up to confirm it.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorCommon Crawl documents a robots.txt opt-out ("User-agent: CCBot / Disallow: /"). The registry lists CCBot as respecting robots.txt.

Copy-Paste robots.txt & Cloudflare WAF Rules for CCBot

robots.txt — allow public pages, keep private routes off-limits

User-agent: CCBot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: CCBot
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "CCBot")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?Indirectly: Common Crawl publishes an open dataset that the registry says is used for machine learning and AI, among other uses
Executes JavaScript?Not documented
Can submit forms or attempt logins?Not documented
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen CCBot? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About CCBot

What is CCBot?

The crawler of the Common Crawl Foundation, which publishes a free open web archive going back to 2008.

How do I block CCBot?

Common Crawl says to add "User-agent: CCBot" and "Disallow: /" to robots.txt.

How can I tell if CCBot is real?

Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS (except IPv6) and publishes them as JSON at index.commoncrawl.org/ccbot.json. It warns that some crawlers falsely claim to be CCBot.

Could This Be a Scam? What to Do

  • Anyone can copy the CCBot User-Agent. Common Crawl Foundation publishes an IP list for it, so compare the visiting address against that list before you trust the visit.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the CCBot or Common Crawl Foundation name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More AI Agents

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Common Crawl Foundation IP feed, Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Common Crawl: CCBot, Common Crawl: ccbot.json IP ranges. FraudMeNot is not affiliated with Common Crawl Foundation. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.