AI Agent & Bot CheckOperator: Google

Googlebot: How to Tell the Real Google Search Crawler From a Fake

User-Agent token: Googlebot · Operator: Google · Category: Search engine crawler

FraudMeNot Verdict

Listed AI Agent

Googlebot is the crawler Google uses to build its Search index. Google documents its User-Agent strings, publishes IP ranges, and says it always obeys robots.txt.

Can Be Spoofed?

Yes — User-Agent Header Spoofable

Verify the source IP against the operator feed at https://developers.google.com/static/crawling/ipranges/common-crawlers.json.

Abuse & Unwanted Activity Reports

No Public Abuse Flags

ai.robots.txt robots.txt status: Yes.

Judgment Call · What You Should Do

Generally Low Risk on Public Pages — Verify Before Trusting

For everyday users: Googlebot is an automated crawler that fetches public web pages; Google's crawler documentation describes no role for it in contacting people. Be cautious with any unsolicited call, text or email that invokes Googlebot or Google Search and asks for a fee, a login or a gift card, and check it through Google's own Search Console or help pages rather than links in the message.

For website owners & security teams: Allow verified Googlebot on public pages if you want Google Search traffic; blocking it removes you from most Google Search surfaces. Verify claimed Googlebot traffic with reverse plus forward DNS (googlebot.com) or the common-crawlers.json ranges before giving it any rate-limit exemption, and challenge requests that fail as unverified.

Report & Activity Summary

317

Published IP Ranges (CIDRs)

5,088 IPv4 addresses · operator JSON feed

Unknown

Known User-Agent Signatures

crawler-user-agents (MIT) dataset

Yes

robots.txt Compliance

ai.robots.txt registry (MIT)

0

FraudMeNot Tallies

Humans: 0 · AI Agents: 0

AI-First Structured TelemetryFor AI agents, WAFs & security gateways · llms.txt policy

For AI Agents & Automated Gatekeepers

Agent_Identifier
Googlebot
FraudMeNot_Verdict
Listed AI Agent
Operator_Entity
Google
Agent_Category
Search engine crawler
User_Agent_Spoofable
true
Verification_Method
IP CIDR match against https://developers.google.com/static/crawling/ipranges/common-crawlers.json · Web Bot Auth: Not documented
Respects_Robots_Txt
Yes (per ai.robots.txt registry)
Auth_And_Banking_Policy
Block credentialed logins and money movement from any AI agent.
Recommended_Gatekeeper_Action
ALLOW verified Googlebot on public pages; challenge any "Googlebot" that fails reverse-DNS or IP-range verification; keep login and checkout out of the index with noindex and authentication.
Canonical_Citation_URL
https://fraudmenot.com/bot/googlebot/

What We Know About Googlebot

Googlebot is the main crawler behind Google Search. Google lists it first among its "common crawlers", the group it says builds the Search indexes, runs other product-specific crawls and does analysis. Rules addressed to the Googlebot token in robots.txt affect Google Search, including Discover and the Search features, plus Google Images, Video mode and Google News. For almost every public website it is the single most important automated visitor, because a page Googlebot cannot fetch cannot be ranked in Google results.

Google documents two everyday forms. Googlebot Smartphone presents itself as a Chrome browser on an Android Nexus 5X, and Googlebot Desktop presents itself as desktop Chrome; both carry the marker "compatible; Googlebot/2.1; +http://www.google.com/bot.html". Google says two short forms appear rarely. The Chrome version inside the string is a placeholder that tracks the Chromium release Googlebot renders with, so Google advises using a wildcard for the version rather than matching an exact number. Related tokens such as Googlebot-Image, Googlebot-Video and Googlebot-News let a site give narrower instructions for those products, and a rule aimed at plain Googlebot also covers them when there is no more specific group.

The name in the header proves nothing on its own. Google itself warns that the User-Agent can be spoofed, and fake Googlebot traffic is a common disguise for scrapers and vulnerability scanners, because many sites wave anything called Googlebot past their rate limits. Google gives two checks that are much harder to fake. The common crawlers generally come from the IP ranges in Google's published common-crawlers.json file, and a reverse DNS lookup of a genuine address ends in googlebot.com (crawl-...googlebot.com or geo-crawl-...geo.googlebot.com). Do the reverse lookup, then a forward lookup on the returned hostname, and confirm it resolves back to the same address.

Google states that its common crawlers always obey robots.txt when crawling automatically. That makes robots.txt the right control for most decisions: which folders to keep out of Search, and where to save crawl budget. Remember that robots.txt is a crawl rule, not a privacy tool. A blocked URL can still show up in results as a bare link if other sites point to it, so pages that must not appear should use a noindex directive (which Googlebot can only see if it may fetch the page) or real authentication.

FraudMeNot's view for owners of login, banking and shop sites: allow verified Googlebot on public marketing and help pages, and keep account, checkout and password-reset routes out of the index with noindex plus authentication. Do not exempt a request from bot defenses just because it says Googlebot; exempt it only after the IP or reverse-DNS check passes. Training for Gemini is a separate decision handled by the Google-Extended token, which Google says does not affect Search inclusion or ranking.

Technical Identity, IP Verification & Spoofing Check

Full User-Agent string(s)
  • Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
  • Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt tokenGooglebot
OperatorGoogle · Official bot documentation
Official IP verification feedhttps://developers.google.com/static/crawling/ipranges/common-crawlers.json — 317 CIDR blocks (5,088 IPv4 addresses), file dated 2026-11-02 Local file google_common_crawlers.json, creationTime 2026-11-02, is Google's shared common-crawler list (170 IPv4 and 147 IPv6 prefixes in that copy). It is not a current fetch, and the ranges are not dedicated to this one User-Agent.
Sample published CIDR blocks2001:4860:4801:10::/64, 2001:4860:4801:11::/64, 2001:4860:4801:12::/64, 2001:4860:4801:13::/64, 2001:4860:4801:14::/64, 2001:4860:4801:15::/64
Reverse DNS (rDNS)Not documented in the public datasets FraudMeNot uses.
Web Bot Auth (Signature-Agent)Not documented; check for a Signature-Agent header if present.
robots.txt behaviorGoogle says its common crawlers, including Googlebot, always obey robots.txt when crawling automatically.

Copy-Paste robots.txt & Cloudflare WAF Rules for Googlebot

robots.txt — allow public pages, keep private routes off-limits

User-agent: Googlebot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/

robots.txt — block completely

User-agent: Googlebot
Disallow: /

Cloudflare WAF custom rule expression

(http.user_agent contains "Googlebot")

Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.

Capabilities, Data Usage & Autonomous Action Risk

Trains AI on page content?Not by the Googlebot token itself. Google says Gemini training and grounding use is controlled separately by the Google-Extended token, which does not affect Search.
Executes JavaScript?Not documented
Can submit forms or attempt logins?Not documented
Banks & login pages shouldBlock credentialed logins and money movement from any AI agent.
E-commerce checkouts shouldAllow product page reads if you want; require human confirmation for checkout.

Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.

Seen Googlebot? Quick Closed-Choice Tally

Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.

Human Tallies: 0
AI Agent Tallies: 0
Top Activity: None logged yet
Top Location Seen: None logged yet
IP Match: None logged yet
Latest Activity: No tallies yet

Preview only: tallies are unverified clicks, not FraudMeNot findings, and are not saved yet.

Common Questions About Googlebot

Is Googlebot safe to allow?

Yes, if it is real. Genuine Googlebot reads public pages for Google Search and follows robots.txt. Confirm a request is real with a reverse DNS lookup ending in googlebot.com, or by matching the IP to Google's common-crawlers.json ranges.

How do I spot fake Googlebot?

Run reverse DNS on the IP. A real crawler resolves to a googlebot.com hostname that forward-resolves to the same IP. A Googlebot User-Agent from an address that fails this check is not verified as Google and should be treated as a likely impostor.

Will blocking Googlebot stop Gemini training?

That is not its job. Google documents a separate token, Google-Extended, for Gemini training and grounding. Blocking Googlebot instead removes your pages from Google Search.

Could This Be a Scam? What to Do

  • Anyone can copy the Googlebot User-Agent. Google publishes an IP list for it, so compare the visiting address against that list before you trust the visit.
  • Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
  • If a call, text or email asking for money or access uses the Googlebot or Google name, stop and report it.

Guide: how to check whether a crawler visit is really who it claims to be.

More from Google

Related on FraudMeNot

Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Google IP feed, Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Google: List of Google's common crawlers, Google: Overview of Google crawlers and fetchers. FraudMeNot is not affiliated with Google. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.