AIWebIndex: what it is and how to block or allow it
User-Agent token: AIWebIndex · Operator: Lyrenth · Category: AI Search Crawlers
FraudMeNot Verdict
Listed AI Agent
AIWebIndex is the crawler of Lyrenth, an index of web pages served to AI systems. Lyrenth says it honors robots.txt on every fetch and does not train foundation models on what it crawls.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
Check the IP against ip-ranges.json, reverse DNS under lyrenth.com, or the Web Bot Auth signature. A request failing all three should be treated as unverified.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Yes.
Judgment Call · What You Should Do
Generally Low Risk on Public Pages — Verify Before Trusting
For everyday users: Lyrenth's index bot reads pages; it has no reason to phone or text anyone. If a message invokes AIWebIndex or Lyrenth and wants a login or money, treat it as a likely scam and don't respond.
For website owners & security teams: Lyrenth publishes IPs, rDNS and signatures, so AIWebIndex can be verified before you allow it. One robots.txt group for AIWebIndex covers both of its user agents. Read the licensing terms before verifying your domain with Lyrenth.
Report & Activity Summary
22
Published IP Ranges (CIDRs)
22 IPv4 addresses · operator JSON feed
1
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Yes
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
AIWebIndex- FraudMeNot_Verdict
- Listed AI Agent
- Operator_Entity
- Lyrenth
- Agent_Category
- AI Search Crawlers
- User_Agent_Spoofable
true- Verification_Method
- IP CIDR match against
https://lyrenth.com/bot/ip-ranges.json· Web Bot Auth: Lyrenth says its requests carry a Web Bot Auth (RFC 9421, Ed25519) signature. - Respects_Robots_Txt
- Yes (per ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- ALLOW or DISALLOW per your AI-index policy; verify by IP, rDNS or signature first.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/aiwebindex/
What We Know About AIWebIndex
AIWebIndex belongs to Lyrenth, which its crawler policy says is operated by Aleksma Ai, Inc. Lyrenth builds a structured index of public pages and serves it to AI agents and developers through an API, so many agents can read one cached copy instead of each fetching your page.
Lyrenth describes two kinds of fetch. Scheduled crawling uses the AIWebIndex/2.0 string. On-demand fills, when a customer asks for a URL the index lacks, use AIWebIndex-Agent/2.0. Lyrenth says a single Disallow for AIWebIndex covers both, and that it fetches robots.txt at most once every 24 hours.
For checking a visit, Lyrenth lists three methods: a Web Bot Auth signature (RFC 9421) checked against its public key directory, its published IP ranges in ip-ranges.json, and forward-confirmed reverse DNS under lyrenth.com. In our view, a request using the name that passes none of these should be treated as unverified.
One point site owners may want to read closely: Lyrenth says verifying your domain in its dashboard also grants it the right to license your canonical content to AI labs, with no revenue share. Crawling alone does not do that, according to the policy; the grant comes from verification.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | AIWebIndex |
| Operator | Lyrenth · Official bot documentation |
| Official IP verification feed | https://lyrenth.com/bot/ip-ranges.json — 22 CIDR blocks (22 IPv4 addresses), file dated 2026-08-16 |
| Sample published CIDR blocks | 46.224.73.201/32, 46.225.147.220/32, 49.13.94.158/32, 138.201.157.96/32, 142.132.225.162/32, 152.53.93.229/32 |
| Reverse DNS (rDNS) | Lyrenth says every crawler IP has forward-confirmed reverse DNS under lyrenth.com. |
| Web Bot Auth (Signature-Agent) | Lyrenth says its requests carry a Web Bot Auth (RFC 9421, Ed25519) signature. |
| robots.txt behavior | Lyrenth says AIWebIndex implements RFC 9309, honors Disallow and Crawl-delay, and that one AIWebIndex rule covers both its scheduled and on-demand fetches. |
Copy-Paste robots.txt & Cloudflare WAF Rules for AIWebIndex
robots.txt — allow public pages, keep private routes off-limits
User-agent: AIWebIndex
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: AIWebIndex
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "AIWebIndex")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Lyrenth says it does not train foundation models on crawled content, but says verifying a domain grants it the right to license that domain's content to AI labs |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen AIWebIndex? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About AIWebIndex
Who operates AIWebIndex?
Lyrenth, which its crawler policy says is run by Aleksma Ai, Inc.
What is AIWebIndex-Agent?
Lyrenth's on-demand fetch, used when a customer requests a URL not yet in its index. Lyrenth says the AIWebIndex robots.txt token covers it too.
Does Lyrenth train AI on my pages?
Lyrenth's policy says it does not train foundation models on crawled content. It also says verifying your domain grants Lyrenth the right to license your content to AI labs, including for payment, and that TDM-reservation or noai/notrain tags exclude a domain from that licensing.
Could This Be a Scam? What to Do
- Anyone can copy the AIWebIndex User-Agent. Lyrenth publishes an IP list for it, so compare the visiting address against that list before you trust the visit.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the AIWebIndex or Lyrenth name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More AI Agents
- AI2Bot — Content is used to train open language models
- CCBot — Provides open crawl dataset, used for many purposes, includi
- Diffbot — AI Data Providers
- OAI-SearchBot — AI Search Crawler (ChatGPT search)
- PerplexityBot — Search result generation
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Lyrenth IP feed, Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Lyrenth: Crawler policy, Lyrenth: IP ranges (JSON). FraudMeNot is not affiliated with Lyrenth. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.