Guide
How to Verify Googlebot and Spot Fake Crawlers
Short answer
Don't trust the name in your logs. Check the IP address. Google warns that "the HTTP user agent string can be spoofed" (Source 2), so anyone can call their bot "Googlebot." To confirm a visit really came from Google, do a reverse DNS lookup on the IP, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, and then do a forward lookup to confirm it points back to the same IP. Or match the IP against Google's published IP lists (Source 1).
Why fake crawlers matter
A fake "Googlebot" is a visitor lying about who it is, and that's reason enough to take a closer look. Google itself frames verification as useful if you're "concerned that spammers or other troublemakers are accessing your site while claiming to be from Google" (Source 1). Site owners often go easy on anything labeled Googlebot so they don't hurt their search ranking. An impostor counts on that.
Our take: a request that says "Googlebot" but fails the checks below can't be verified as Google. Treat it like any other unknown visitor under your normal security and rate-limit rules. Don't block the real Googlebot by mistake, though. Verify first.
How to verify Googlebot step by step (reverse DNS)
Google's documented method for one-off checks (Source 1):
- Reverse lookup the IP from your logs with the
hostcommand. - Check the domain. It must end in
googlebot.com,google.comorgoogleusercontent.com. - Forward lookup that hostname with
host. - Confirm it returns the original IP. If it doesn't match, it isn't verified.
Google's own example (Source 1): ``` host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1 ```
Why not stop after step 2? Whoever controls an IP address controls its reverse DNS record, so an impostor can make it claim a Google-looking name. The forward lookup in step 3 asks Google's DNS, which the impostor doesn't control, and the IP has to match. That's why both directions matter.
What the hostname should look like
Google groups its crawlers into three categories, each with its own hostname pattern (Source 1):
| Type | Example | Reverse DNS pattern | Respects robots.txt? |
|---|---|---|---|
| Common crawlers | Googlebot | crawl-…googlebot.com or geo-crawl-…geo.googlebot.com | Google says yes, for automatic crawls |
| Special-case crawlers | AdsBot | rate-limited-proxy-…google.com | May or may not |
| User-triggered fetchers | Google Site Verifier | …gae.googleusercontent.com or google-proxy-…google.com | No, because a user requested the fetch |
So a real Google request can ignore robots.txt (a user-triggered fetcher, for example). Ignoring robots.txt doesn't by itself prove a visitor is fake. The IP check does.
Which Googlebot variants show up in logs?
"Googlebot" is really a family of crawlers, and each one has its own user agent token (Source 2). Knowing the names helps you spot a label that doesn't exist. Google lists, among others:
- Googlebot Smartphone and Googlebot Desktop. Both use the token Googlebot. Rules for that token affect Google Search, including Discover, plus Google Images, Video mode and Google News (Source 2).
- Googlebot-Image. Affects Google Images, Discover, Video mode and Search features that show images, logos and favicons (Source 2).
- Googlebot-Video. Affects video-related Search features and other products that depend on videos (Source 2).
- Googlebot-News. Has no separate user agent string of its own; it crawls with the regular Googlebot strings. Its rules affect news.google.com and the Google News app (Source 2).
- Storebot-Google. Affects every surface of Google Shopping, such as the Shopping tab in Search (Source 2).
Google adds that the list "is not exhaustive" and covers the requestors most likely to appear in log files (Source 2). The real Chrome-based strings also carry a changing browser version number, so an exact-match filter on the whole string can break. Our take: match on the token, then confirm with DNS or the IP lists before you trust it.
Googlebot IP ranges: match against Google's lists
For lots of lookups, compare IPs against Google's published JSON lists instead of running DNS on each one (Source 1). Google publishes separate lists for common crawlers, special crawlers, user-triggered fetchers and user-triggered agents. The ranges are in CIDR format. Other Google traffic, such as Apps Script, is covered by Google's general IP list (Source 1). Load the lists from Google's documentation page rather than copying ranges into a blog post or config by hand, because they change.
Other crawlers
Some, such as OpenAI, publish their own IP lists, so the same idea applies: check the vendor's docs, then match the IP. For example, OpenAI publishes separate IP lists for OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot (Source 3). OpenAI also says that because ChatGPT-User actions are initiated by a user, "robots.txt rules may not apply" (Source 3). See our AI agent and bot directory for crawler-by-crawler notes, including /bot/chatgpt-user/, /bot/gptbot/ and /bot/bytespider/. If a fake crawler is part of a scam, see /report/.
Verifying crawlers is one layer. Our take: it won't stop every bad bot, and lots of unwanted traffic doesn't pretend to be anyone.
FAQ
Can the Googlebot user agent be faked? Yes. Google says the user agent string can be spoofed (Source 2).
How do I verify Googlebot? Reverse DNS the IP, confirm the hostname is in googlebot.com, google.com or googleusercontent.com, then forward DNS it back to the same IP. Or match the IP against Google's published lists (Source 1).
Why isn't reverse DNS alone enough? The owner of an IP sets its reverse record and can fake a Google-looking name. The forward lookup confirms the name really points back to that IP.
Does real Googlebot obey robots.txt? Google says its common crawlers, Googlebot included, always respect robots.txt for automatic crawls. Some special-case crawlers and user-triggered fetchers may not (Source 1).
Do other bots publish IP lists? Some do; OpenAI publishes lists for its crawlers (Source 3). Check each vendor's documentation.
Sources (all checked 2026-10-24)
- Google for Developers, "Verify requests from Google crawlers and fetchers." https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests — accessed 2026-10-24
- Google for Developers, "List of Google's common crawlers." https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers — accessed 2026-10-24
- OpenAI, "Overview of OpenAI Crawlers." https://developers.openai.com/api/docs/bots — accessed 2026-10-24
Google, Googlebot, OpenAI and GPTBot are trademarks of their owners, used only to identify them. FraudMeNot is not affiliated with or endorsed by Google or OpenAI.