AI Agent & Bot CheckOperator: Diffbot

Diffbot-User: what it is and how to block or allow it

User-Agent token: Diffbot-User · Operator: Diffbot · Category: AI Assistants

FraudMeNot Verdict

AI Agent (User-Triggered)

Diffbot-User is the agent Diffbot uses for requests made on behalf of a person browsing a URL with Diffbot software, as opposed to its proactive Diffbot crawler.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

Diffbot publishes no IP list or full UA string for Diffbot-User, so the name alone cannot prove origin.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Yes.

Judgment Call · What You Should Do

Generally Low Risk on Public Pages — Verify Before Trusting

For everyday users: Diffbot-User loads a page for someone using Diffbot tools; it does not message the site's owner or visitors. Login or payment requests in its name are likely fraud.

For website owners & security teams: Diffbot-User acts for a person using Diffbot software. A robots.txt group expresses your preference; Diffbot says agreements can override it. Keep logins behind human checks.

Report & Activity Summary

None

Published IP Ranges (CIDRs)

No official IP list

1

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Partially

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
Diffbot-User
FraudMeNot_Verdict
AI Agent (User-Triggered)
Operator_Entity
Diffbot
Agent_Category
AI Assistants
User_Agent_Spoofable
true
Verification_Method
None available (no operator IP list or signing key) · Web Bot Auth: Not documented
Respects_Robots_Txt
Partially (user-triggered fetches may ignore robots.txt; ai.robots.txt registry lists: Yes)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
ALLOW read access if you are comfortable with user-driven Diffbot fetches; require human step-up on login and payment.
Canonical_Citation_URL
https://fraudmenot.com/bot/diffbot-user/

What We Know About Diffbot-User

Diffbot documents two user agents in its robots.txt FAQ. Diffbot is its proactive crawler for its Knowledge Graph and web search, which it says is not used for AI training. Diffbot-User is used for requests originating on behalf of a human user browsing a URL with Diffbot software, in response to their input.

Diffbot says its crawls follow robots.txt by default, including Disallow and Crawl-delay. It also says that in specific cases, typically a partnership or agreement with the site, the robots.txt instruction can be overridden. robots.txt is a voluntary standard (RFC 9309), not an access-control or legal mechanism; a fetcher that skips it is not, by that alone, doing anything wrong.

Diffbot says it does not crawl pages for training generative AI foundation models, and that crawling uses caching, compression and conditional requests to limit server load.

Diffbot's FAQ names the Diffbot-User token but does not give a full user-agent string or an IP list. In our view, a visit carrying the name should be treated as a claim until you can tie it to a known Diffbot use.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • Diffbot-User
robots.txt tokenDiffbot-User
OperatorDiffbot · Official bot documentation
Official IP verification feedNone published by the operator
Sample published CIDR blocksNone published
Reverse DNS (rDNS)Not documented in the public datasets FraudMeNot uses.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorDiffbot says its crawls follow robots.txt by default, including Disallow and Crawl-delay, but can be overridden where a site has an agreement.

Copy-Paste robots.txt & Cloudflare WAF Rules for Diffbot-User

robots.txt — allow public pages, keep private routes off-limits

User-agent: Diffbot-User
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: Diffbot-User
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "Diffbot-User")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?No, per Diffbot, which says it does not crawl pages for training generative AI foundation models
Executes JavaScript?Not documented
Can submit forms or attempt logins?May act for a person browsing with Diffbot software; treat form and login attempts as untrusted.
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen Diffbot-User? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About Diffbot-User

What is the difference between Diffbot and Diffbot-User?

Diffbot crawls proactively for its Knowledge Graph and search. Diffbot-User fetches on behalf of a person using Diffbot software.

Does Diffbot train AI on my pages?

Diffbot says it does not crawl pages for training generative AI foundation models.

Can robots.txt be overridden?

Diffbot says yes in specific cases, typically when there is an agreement with the site.

Could This Be a Scam? What to Do

  • Anyone can copy the Diffbot-User User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the Diffbot-User or Diffbot name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More from Diffbot

  • Diffbot — AI Data Providers
  • omgilibot — Data is sold
  • CCBot — Provides open crawl dataset, used for many purposes, includi
  • ChatGPT-User — AI Assistant · User-Triggered Fetcher

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Diffbot Docs: Does Diffbot respect robots.txt?. FraudMeNot is not affiliated with Diffbot. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.