omgilibot: what it is and how to block or allow it
User-Agent token: omgilibot · Operator: Webz.io · Category: Data is sold
FraudMeNot Verdict
Listed AI Agent
omgilibot is the crawler behind Webz.io, a web-data service sold to media monitors, researchers and developers. In a 2017 blog post, Webz.io said it honors robots.txt requests to stop crawling.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
Webz.io's 2017 post gives a user agent but no addresses, so verification by IP is not possible.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Yes.
Judgment Call · What You Should Do
Generally Low Risk on Public Pages — Verify Before Trusting
For everyday users: An email in its name selling "removal" for a fee is likely a scam; Webz.io says you can opt out via robots.txt or by contacting it directly.
For website owners & security teams: Data crawled by omgilibot is resold through the Webz.io API. To opt out, add robots.txt groups for both omgilibot and omgili, since the string Webz.io documents is omgili.
Report & Activity Summary
None
Published IP Ranges (CIDRs)
No official IP list
1
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Yes
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
omgilibot- FraudMeNot_Verdict
- Listed AI Agent
- Operator_Entity
- Webz.io
- Agent_Category
- Data is sold
- User_Agent_Spoofable
true- Verification_Method
- None available (no operator IP list or signing key) · Web Bot Auth: Not documented
- Respects_Robots_Txt
- Yes (per ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- DISALLOW omgilibot and omgili if you do not want your pages in Webz.io data feeds.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/omgilibot/
What We Know About omgilibot
Webz.io wrote about its own crawler in a 2017 blog post. It said the Omgili bot was built to power the now-discontinued Omgili search engine and then fed Webz.io, a web-crawling service. The user agent it gave then was omgili/0.5 +https://omgili.com.
Webz.io says the bot focuses on news sites, blogs, reviews, ecommerce and message boards, and that crawled pages are indexed and made available through the Webz.io API to companies, researchers and developers. It names social-media tools such as Hootsuite and Sprinklr as services that rely on that data.
On opting out, Webz.io says you can tell it directly or use robots.txt, and that it complies with those requests. Note that the ai.robots.txt registry lists the token as omgilibot while the string Webz.io shows is omgili; adding groups for both is the cautious choice.
The post is from 2017, and we found no IP list or reverse-DNS method from Webz.io for this bot. Treat a visit using the name as a claim, and judge it by behavior and volume.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | omgilibot |
| Operator | Webz.io · Official bot documentation |
| Official IP verification feed | None published by the operator |
| Sample published CIDR blocks | None published |
| Reverse DNS (rDNS) | Not documented in the public datasets FraudMeNot uses. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | Webz.io says the Omgili bot complies with robots.txt opt-out requests. |
Copy-Paste robots.txt & Cloudflare WAF Rules for omgilibot
robots.txt — allow public pages, keep private routes off-limits
User-agent: omgilibot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: omgilibot
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "omgilibot")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Not stated by Webz.io; it sells the crawled data through its API |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen omgilibot? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About omgilibot
Who runs omgilibot?
Webz.io, which says the bot once powered the Omgili search engine.
What happens to crawled pages?
Webz.io says they are indexed and made available through its API.
Which token should I block?
Webz.io's post shows omgili/0.5; the registry uses omgilibot. Block both to be safe.
Could This Be a Scam? What to Do
- Anyone can copy the omgilibot User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the omgilibot or Webz.io name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More from Webz.io
- omgili — Data is sold
- Webzio-Extended — AI Data Scrapers
- Diffbot — AI Data Providers
- CCBot — Provides open crawl dataset, used for many purposes, includi
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Webz.io: What is the Omgili Bot?. FraudMeNot is not affiliated with Webz.io. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.