AI Agent & Bot CheckOperator: Scrapy (open-source, maintained by Zyte)

Scrapy

User-Agent token: Scrapy · Operator: Scrapy (open-source, maintained by Zyte) · Category: Scrapes data for a variety of uses including training AI

FraudMeNot Verdict

AI Data Collection Bot

Scrapy is an open-source Python framework for crawling sites and extracting data, maintained by Zyte. A Scrapy User-Agent means someone ran it, not that Zyte did.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

No official IP list is published, so the User-Agent alone cannot prove the request is real.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Unclear.

Judgment Call · What You Should Do

Your Choice — Your Content May Be Collected for AI

For everyday users: Scrapy is a programming tool, not a company that contacts people. Nobody sends texts or emails as "Scrapy", so treat any such message as fake.

For website owners & security teams: A Scrapy User-Agent is a third party running the open-source framework. Whether robots.txt is honored depends on that person's settings. Rate-limit or challenge the default User-Agent; use bot management for crawlers that disguise themselves.

Report & Activity Summary

None

Published IP Ranges (CIDRs)

No official IP list

1

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Unclear

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
Scrapy
FraudMeNot_Verdict
AI Data Collection Bot
Operator_Entity
Scrapy (open-source, maintained by Zyte)
Agent_Category
Scrapes data for a variety of uses including training AI
User_Agent_Spoofable
true
Verification_Method
None available (no operator IP list or signing key) · Web Bot Auth: Not documented
Respects_Robots_Txt
Unclear (not stated by the operator or the ai.robots.txt registry)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
RATE-LIMIT or CHALLENGE the default Scrapy User-Agent; there is no central operator to verify.
Canonical_Citation_URL
https://fraudmenot.com/bot/scrapy/

What We Know About Scrapy

Scrapy is not a single bot. It is a popular open-source Python framework for crawling websites and extracting structured data. Scrapy's website says it is maintained by Zyte, has been in production for more than 15 years, and has hundreds of contributors. The registry quotes a Zyte line that AI and machine learning applications often need large amounts of data, and web data extraction is an efficient way to build it.

Because anyone can install Scrapy, a request with the default "Scrapy/x.y (+https://scrapy.org)" User-Agent could come from a student project, a price monitor, a researcher or a data broker.

The registry lists Scrapy's robots.txt status as unclear, which makes sense: Scrapy has a setting for obeying robots.txt, and whether it is on depends on the person running the crawl. There is no IP list, because there is no central operator.

Our take: the default Scrapy User-Agent is a reasonable thing to rate-limit or challenge on sites that do not offer public data. Blocking it will not stop determined scrapers who change their headers.

Reading your logs: the default string "Scrapy/x.y (+https://scrapy.org)" points to the project's site, not to the person running the crawl, so it tells you the tool and not the owner. A crawl that ignores your robots.txt shows a choice made in that person's settings, not by Zyte. Look at request rate and paths to judge intent.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • Scrapy/1.0.3 (+http://scrapy.org)
robots.txt tokenScrapy
OperatorScrapy (open-source, maintained by Zyte) · Official bot documentation
Official IP verification feedNone published by the operator
Sample published CIDR blocksNone published
Reverse DNS (rDNS)Not documented in the public datasets FraudMeNot uses.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorWhether a Scrapy crawl follows robots.txt depends on the person running it. The registry lists its status as unclear.

Copy-Paste robots.txt & Cloudflare WAF Rules for Scrapy

robots.txt — allow public pages, keep private routes off-limits

User-agent: Scrapy
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: Scrapy
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "Scrapy")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?Depends on who runs it; the registry notes Zyte promotes web data for AI and machine learning
Executes JavaScript?Not documented
Can submit forms or attempt logins?Not documented
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen Scrapy? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About Scrapy

What is Scrapy?

An open-source Python framework for crawling websites and extracting data, maintained by Zyte, per scrapy.org.

Is Scrapy traffic from Zyte?

Not necessarily. Anyone can run Scrapy; the User-Agent only shows which tool was used.

Does Scrapy obey robots.txt?

It depends on the user's settings. The ai.robots.txt registry lists its status as unclear.

Could This Be a Scam? What to Do

  • Anyone can copy the Scrapy User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the Scrapy or Scrapy (open-source, maintained by Zyte) name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More AI Agents

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Scrapy. FraudMeNot is not affiliated with Scrapy (open-source, maintained by Zyte). Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.