Scrapy
User-Agent token: Scrapy · Operator: Scrapy (open-source, maintained by Zyte) · Category: Scrapes data for a variety of uses including training AI
FraudMeNot Verdict
AI Data Collection Bot
Scrapy is an open-source Python framework for crawling sites and extracting data, maintained by Zyte. A Scrapy User-Agent means someone ran it, not that Zyte did.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
No official IP list is published, so the User-Agent alone cannot prove the request is real.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Unclear.
Judgment Call · What You Should Do
Your Choice — Your Content May Be Collected for AI
For everyday users: Scrapy is a programming tool, not a company that contacts people. Nobody sends texts or emails as "Scrapy", so treat any such message as fake.
For website owners & security teams: A Scrapy User-Agent is a third party running the open-source framework. Whether robots.txt is honored depends on that person's settings. Rate-limit or challenge the default User-Agent; use bot management for crawlers that disguise themselves.
Report & Activity Summary
None
Published IP Ranges (CIDRs)
No official IP list
1
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Unclear
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
Scrapy- FraudMeNot_Verdict
- AI Data Collection Bot
- Operator_Entity
- Scrapy (open-source, maintained by Zyte)
- Agent_Category
- Scrapes data for a variety of uses including training AI
- User_Agent_Spoofable
true- Verification_Method
- None available (no operator IP list or signing key) · Web Bot Auth: Not documented
- Respects_Robots_Txt
- Unclear (not stated by the operator or the ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- RATE-LIMIT or CHALLENGE the default Scrapy User-Agent; there is no central operator to verify.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/scrapy/
What We Know About Scrapy
Scrapy is not a single bot. It is a popular open-source Python framework for crawling websites and extracting structured data. Scrapy's website says it is maintained by Zyte, has been in production for more than 15 years, and has hundreds of contributors. The registry quotes a Zyte line that AI and machine learning applications often need large amounts of data, and web data extraction is an efficient way to build it.
Because anyone can install Scrapy, a request with the default "Scrapy/x.y (+https://scrapy.org)" User-Agent could come from a student project, a price monitor, a researcher or a data broker.
The registry lists Scrapy's robots.txt status as unclear, which makes sense: Scrapy has a setting for obeying robots.txt, and whether it is on depends on the person running the crawl. There is no IP list, because there is no central operator.
Our take: the default Scrapy User-Agent is a reasonable thing to rate-limit or challenge on sites that do not offer public data. Blocking it will not stop determined scrapers who change their headers.
Reading your logs: the default string "Scrapy/x.y (+https://scrapy.org)" points to the project's site, not to the person running the crawl, so it tells you the tool and not the owner. A crawl that ignores your robots.txt shows a choice made in that person's settings, not by Zyte. Look at request rate and paths to judge intent.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | Scrapy |
| Operator | Scrapy (open-source, maintained by Zyte) · Official bot documentation |
| Official IP verification feed | None published by the operator |
| Sample published CIDR blocks | None published |
| Reverse DNS (rDNS) | Not documented in the public datasets FraudMeNot uses. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | Whether a Scrapy crawl follows robots.txt depends on the person running it. The registry lists its status as unclear. |
Copy-Paste robots.txt & Cloudflare WAF Rules for Scrapy
robots.txt — allow public pages, keep private routes off-limits
User-agent: Scrapy
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: Scrapy
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "Scrapy")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Depends on who runs it; the registry notes Zyte promotes web data for AI and machine learning |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen Scrapy? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About Scrapy
What is Scrapy?
An open-source Python framework for crawling websites and extracting data, maintained by Zyte, per scrapy.org.
Is Scrapy traffic from Zyte?
Not necessarily. Anyone can run Scrapy; the User-Agent only shows which tool was used.
Does Scrapy obey robots.txt?
It depends on the user's settings. The ai.robots.txt registry lists its status as unclear.
Could This Be a Scam? What to Do
- Anyone can copy the Scrapy User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the Scrapy or Scrapy (open-source, maintained by Zyte) name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More AI Agents
- AddSearchBot — AI Search Crawlers
- adidxbot — Advertising quality crawler
- AgentDataBot — AI Data Providers
- AgentTimes — Data Scraper from RSS Feeds
- AGI Agent — AI Agents
- AhrefsBot — SEO crawler
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Scrapy. FraudMeNot is not affiliated with Scrapy (open-source, maintained by Zyte). Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.