AI Agent & Bot CheckOperator: Crawlspace

Crawlspace: what it is and how to block or allow it

User-Agent token: Crawlspace · Operator: Crawlspace · Category: AI Data Providers

FraudMeNot Verdict

Listed AI Agent

Crawlspace is a hosted platform where developers deploy their own web crawlers. Crawlspace says every crawler on it sends a user agent starting with Crawlspace, and follows robots.txt by default.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

Because many developers share the Crawlspace prefix and no address list exists, the header tells you the platform, not the person behind the crawl.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Yes.

Judgment Call · What You Should Do

Generally Low Risk on Public Pages — Verify Before Trusting

For everyday users: Crawlspace is hosting for other people's crawlers. A note claiming to be "Crawlspace support" that asks for your hosting password is probably phishing.

For website owners & security teams: Crawlspace says one robots.txt group for Crawlspace covers every crawler on its platform. robots.txt is voluntary (RFC 9309); add a WAF rule on the prefix for an enforced block.

Report & Activity Summary

None

Published IP Ranges (CIDRs)

No official IP list

1

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Yes

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
Crawlspace
FraudMeNot_Verdict
Listed AI Agent
Operator_Entity
Crawlspace
Agent_Category
AI Data Providers
User_Agent_Spoofable
true
Verification_Method
None available (no operator IP list or signing key) · Web Bot Auth: Not documented
Respects_Robots_Txt
Yes (per ai.robots.txt registry)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
DISALLOW Crawlspace if you do not want third-party developer crawlers; one token covers all.
Canonical_Citation_URL
https://fraudmenot.com/bot/crawlspace/

What We Know About Crawlspace

Crawlspace is not one crawler but a platform. Its site describes a centralized service where developers build and deploy crawlers to gather data for apps and AI agents, with LLM or selector-based extraction.

Crawlspace says developers cannot change the User-Agent header. The platform sets it, includes the author's UUID, and prefixes it with the string Crawlspace. That gives site owners one token to block every crawler on the platform.

Crawlspace's own example robots.txt is User-Agent: Crawlspace with Disallow: /. It says crawlers on the platform are polite by default, following robots.txt and rate-limiting responses, and that a shared cache serves repeat requests so they do not reach your server.

Crawlspace publishes no IP list on its site, and different developers can deploy different crawlers, so in our view volume and behavior are the better signals than the name.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • Crawlspace<author-UUID> (prefix "Crawlspace", set by the platform)
robots.txt tokenCrawlspace
OperatorCrawlspace · Official bot documentation
Official IP verification feedNone published by the operator
Sample published CIDR blocksNone published
Reverse DNS (rDNS)Not documented in the public datasets FraudMeNot uses.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorCrawlspace says crawlers on its platform follow robots.txt and rate-limiting responses by default; its example blocks all of them with User-Agent: Crawlspace.

Copy-Paste robots.txt & Cloudflare WAF Rules for Crawlspace

robots.txt — allow public pages, keep private routes off-limits

User-agent: Crawlspace
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: Crawlspace
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "Crawlspace")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?Not stated; Crawlspace says developers use it to gather data for apps and agents
Executes JavaScript?Not documented
Can submit forms or attempt logins?Not documented
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen Crawlspace? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About Crawlspace

Is Crawlspace one company's crawler?

No. It is a platform where developers deploy their own crawlers.

Can I block all of them at once?

Crawlspace says yes: User-Agent: Crawlspace with Disallow: /.

Can developers hide the Crawlspace name?

Crawlspace says the User-Agent is set by the platform and cannot be changed.

Could This Be a Scam? What to Do

  • Anyone can copy the Crawlspace User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the Crawlspace or Crawlspace name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More AI Agents

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Crawlspace: FAQ. FraudMeNot is not affiliated with Crawlspace. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.