Crawl4AI
User-Agent token: Crawl4AI · Operator: Unknown operator · Category: Undocumented AI Agents
FraudMeNot Verdict
Listed AI Agent
Description unavailable from knownagents.com
Can Be Spoofed?
Yes — User-Agent Header Spoofable
No official IP list is published, so the User-Agent alone cannot prove the request is real.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Unclear.
Judgment Call · What You Should Do
Generally Low Risk on Public Pages — Verify Before Trusting
For everyday users: Everyday users do not need to do anything about Crawl4AI: it reads web pages, and it is not a person contacting you. Never give any AI agent your bank username, password, or one-time codes. Any call, text, or email claiming to be from "Crawl4AI" asking you to act is a scam.
For website owners & security teams: Crawl4AI is run by an operator the registry does not name. Decide whether to allow it on public pages; robots.txt status per the registry: Unclear. Keep login, checkout, and payment routes behind human verification regardless of User-Agent.
Report & Activity Summary
None
Published IP Ranges (CIDRs)
No official IP list
1
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Unclear
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
Crawl4AI- FraudMeNot_Verdict
- Listed AI Agent
- Operator_Entity
- Unknown operator
- Agent_Category
- Undocumented AI Agents
- User_Agent_Spoofable
true- Verification_Method
- None available (no operator IP list or signing key) · Web Bot Auth: Not documented
- Respects_Robots_Txt
- Unclear (not stated by the operator or the ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- ALLOW or DISALLOW per your AI policy; always require human step-up on login and payment routes.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/crawl4ai/
What We Know About Crawl4AI
Security advisories (NVD, checked Nov 2, 2026): Crawl4AI is open-source crawler software that people run themselves, so who is behind a request depends on who deployed it. NVD lists several flaws fixed in Crawl4AI 0.9.5, published Oct 29, 2026 and deferred for NVD analysis, including CVE-2026-119118 (improper certificate validation because Chromium always launches with certificate errors ignored), CVE-2026-119119 (denial of service in the Docker API server) and CVE-2026-119120 (untrusted model loading). The records link the project's GitHub security advisories. Anyone running Crawl4AI should move to 0.9.5 or later.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | Crawl4AI |
| Operator | Unknown operator · Official bot documentation |
| Official IP verification feed | None published by the operator |
| Sample published CIDR blocks | None published |
| Reverse DNS (rDNS) | Not documented in the public datasets FraudMeNot uses. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | The ai.robots.txt registry lists robots.txt respect as: Unclear. |
Copy-Paste robots.txt & Cloudflare WAF Rules for Crawl4AI
robots.txt — allow public pages, keep private routes off-limits
User-agent: Crawl4AI
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: Crawl4AI
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "Crawl4AI")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Not stated by the registry |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen Crawl4AI? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About Crawl4AI
What is Crawl4AI?
Description unavailable from knownagents.com
Does Crawl4AI respect robots.txt?
The open-source ai.robots.txt registry lists its robots.txt status as: Unclear.
How do I block Crawl4AI?
Add "User-agent: Crawl4AI" and "Disallow: /" to robots.txt, and add a firewall rule on the User-Agent for an enforced block.
Could This Be a Scam? What to Do
- Anyone can copy the Crawl4AI User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the Crawl4AI or Unknown operator name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More from Unknown operator
- AddSearchBot — AI Search Crawlers
- AgentDataBot — AI Data Providers
- Aranet-SearchBot — Undocumented AI Agents
- BixelBot — AI Data Providers
- Channel3Bot — AI Search Crawlers
- ChatGLM-Spider — AI Data Scrapers
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), NVD: CVE-2026-119118, Crawl4AI GitHub advisory GHSA-j2qh-f73r-mg3q. FraudMeNot is not affiliated with Unknown operator. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.