AI Agent & Bot CheckOperator: Diffbot

Diffbot

User-Agent token: Diffbot · Operator: Diffbot · Category: AI Data Providers

FraudMeNot Verdict

Listed AI Agent

Diffbot extracts and structures website content with AI-based visual understanding. Its data feeds a knowledge graph and APIs used to ground AI answers.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

No official IP list is published, so the User-Agent alone cannot prove the request is real.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Unclear.

Judgment Call · What You Should Do

Generally Low Risk on Public Pages — Verify Before Trusting

For everyday users: Diffbot is a data company's crawler. It does not contact consumers. Ignore messages using its name to ask for payment or logins.

For website owners & security teams: Diffbot structures page content into data that its customers use, including for AI. Disallow it in robots.txt and block the token at the WAF if you do not want that. Its robots.txt status is unclear in the registry.

Report & Activity Summary

None

Published IP Ranges (CIDRs)

No official IP list

1

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Unclear

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
Diffbot
FraudMeNot_Verdict
Listed AI Agent
Operator_Entity
Diffbot
Agent_Category
AI Data Providers
User_Agent_Spoofable
true
Verification_Method
None available (no operator IP list or signing key) · Web Bot Auth: Not documented
Respects_Robots_Txt
Unclear (not stated by the operator or the ai.robots.txt registry)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
ALLOW or DISALLOW per your data-reuse policy; BLOCK the token at the WAF for an enforced stop.
Canonical_Citation_URL
https://fraudmenot.com/bot/diffbot/

What We Know About Diffbot

Diffbot is a company that turns web pages into structured data. The ai.robots.txt registry says its crawler extracts and structures content using AI-powered visual understanding, and provides knowledge graph data for applications such as market intelligence. Diffbot's home page now pitches grounding AI on verified facts from the public web, and a web search API.

The User-Agent in our dataset is an old Firefox 3.5 string with "Diffbot/0.1; +http://www.diffbot.com" inside it. The registry lists its robots.txt status as unclear. Diffbot's documentation has a help topic titled "Does Diffbot respect robots.txt?", but we could not read its answer in our two checks, so we do not state one here.

Diffbot customers can also run crawls of their own choosing through its Crawl API. That means a visit may reflect a specific customer's job rather than a general index crawl.

Our take: Diffbot data is resold and reused, so allowing it spreads your content further. If that is unwanted, add a robots.txt disallow and a WAF rule on the token. FraudMeNot found no Diffbot IP list.

Reading your logs: a Firefox 3.5 string is very old, so a Diffbot hit can look odd next to modern browsers. The Diffbot/0.1 marker inside it is what identifies the crawler. Since Crawl API customers choose their own targets, a sudden burst on one section of your site may be a single customer's job rather than a routine index pass.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
robots.txt tokenDiffbot
OperatorDiffbot · Official bot documentation
Official IP verification feedNone published by the operator
Sample published CIDR blocksNone published
Reverse DNS (rDNS)Not documented in the public datasets FraudMeNot uses.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorThe ai.robots.txt registry lists robots.txt respect as: Unclear.

Copy-Paste robots.txt & Cloudflare WAF Rules for Diffbot

robots.txt — allow public pages, keep private routes off-limits

User-agent: Diffbot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: Diffbot
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "Diffbot")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?Not stated by the registry
Executes JavaScript?Not documented
Can submit forms or attempt logins?Not documented
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen Diffbot? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About Diffbot

What is Diffbot?

A company whose crawler extracts and structures web content. The registry says it provides knowledge graph data for applications like market intelligence.

Does Diffbot respect robots.txt?

The ai.robots.txt registry lists its status as unclear. Use a firewall rule if you need a hard block.

Why does Diffbot use an old Firefox User-Agent?

FraudMeNot does not know. The dataset string includes Firefox 3.5 and "Diffbot/0.1"; match on the Diffbot token, not the browser version.

Could This Be a Scam? What to Do

  • Anyone can copy the Diffbot User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the Diffbot or Diffbot name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More from Diffbot

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Diffbot. FraudMeNot is not affiliated with Diffbot. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.