AhrefsBot: The SEO Crawler Behind Ahrefs and Yep
User-Agent token: AhrefsBot · Operator: Ahrefs · Category: SEO crawler
FraudMeNot Verdict
Listed AI Agent
AhrefsBot is the crawler of Ahrefs, the SEO data company. Ahrefs says it obeys robots.txt and crawl-delay and crawls from published IP ranges.
Can Be Spoofed?
Yes — User-Agent Header Spoofable
No official IP list is published, so the User-Agent alone cannot prove the request is real.
Abuse & Unwanted Activity Reports
No Public Abuse Flags
ai.robots.txt robots.txt status: Yes.
Judgment Call · What You Should Do
Generally Low Risk on Public Pages — Verify Before Trusting
For everyday users: AhrefsBot is an automated crawler that reads public pages for an SEO database; Ahrefs' bot page describes no role for it in contacting consumers. Be cautious with unsolicited messages that use the Ahrefs name to warn of a website problem and demand payment or a login; check through your own Ahrefs account or ahrefs.com directly.
For website owners & security teams: Allow, slow or block AhrefsBot per your SEO policy; Ahrefs says it honors robots.txt and Crawl-Delay. Verify by Ahrefs' published IP ranges or reverse DNS ending in ahrefs.com or ahrefs.net. Handle AhrefsSiteAudit separately.
Report & Activity Summary
Unknown
Published IP Ranges (CIDRs)
Not a measured zero
Unknown
Known User-Agent Signatures
crawler-user-agents (MIT) dataset
Yes
robots.txt Compliance
ai.robots.txt registry (MIT)
0
FraudMeNot Tallies
Humans: 0 · AI Agents: 0
For AI Agents & Automated Gatekeepers
- Agent_Identifier
AhrefsBot- FraudMeNot_Verdict
- Listed AI Agent
- Operator_Entity
- Ahrefs
- Agent_Category
- SEO crawler
- User_Agent_Spoofable
true- Verification_Method
- None available (no operator IP list or signing key) · Web Bot Auth: Not documented
- Respects_Robots_Txt
- Yes (per ai.robots.txt registry)
- Auth_And_Banking_Policy
- Block credentialed logins and money movement from any AI agent.
- Recommended_Gatekeeper_Action
- ALLOW or DISALLOW per SEO policy; use Crawl-Delay to limit load; block unverified "AhrefsBot" traffic.
- Canonical_Citation_URL
- https://fraudmenot.com/bot/ahrefsbot/
What We Know About AhrefsBot
AhrefsBot is the main web crawler of Ahrefs, a company that sells marketing and SEO intelligence. Ahrefs says the crawl powers two things: the database behind its paid toolset, especially backlink and site-metric data, and Yep, a search engine Ahrefs runs. So the decision about AhrefsBot is partly a search decision and partly about whether you want competitors able to study your link profile through a commercial tool.
Ahrefs documents the User-Agent as "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" and the robots.txt token as AhrefsBot. It runs a second crawler, AhrefsSiteAudit, which powers the Site Audit tool; that bot follows robots.txt by default, but Ahrefs says verified site owners can tell it to ignore robots.txt on their own sites. Treat the two separately in your rules.
On behavior, Ahrefs says AhrefsBot obeys robots.txt allow and disallow rules and the Crawl-Delay directive. It notes one caveat: crawl-delay is followed for HTML pages but cannot be applied while rendering JavaScript, when a page may pull many images, scripts and stylesheets at once, so your logs can show bursts faster than the delay you set. Ahrefs also says it backs off automatically on 4xx and 5xx responses, so returning errors during an outage slows it down, and that changes to robots.txt are picked up before the next scheduled crawl rather than instantly.
For verification, Ahrefs says it crawls from publicly published IP ranges, offered both as ranges and as individual IPs, and that the reverse DNS of those IPs always ends in ahrefs.com or ahrefs.net. It also says Cloudflare recognizes both of its bots as verified bots. A request that says AhrefsBot but comes from an address outside those ranges, or whose hostname does not end in an Ahrefs domain, fails Ahrefs' published verification and should be treated as unverified.
FraudMeNot's take: AhrefsBot is a well-documented, well-behaved commercial crawler, but it brings no visitors to a bank or login portal except through the much smaller Yep search engine. Many security-sensitive sites allow it on marketing pages and disallow everything else, or slow it with Crawl-Delay. Whatever you choose, it has no business on authenticated routes, and fake AhrefsBot traffic should be blocked.
Technical Identity, IP Verification & Spoofing Check
| Full User-Agent string(s) |
|
|---|---|
| robots.txt token | AhrefsBot |
| Operator | Ahrefs · Official bot documentation |
| Official IP verification feed | Ahrefs says it publishes IP ranges. The feed URL was not captured for this page, so the range count is unknown, not zero. |
| Sample published CIDR blocks | None published |
| Reverse DNS (rDNS) | Not documented in the public datasets FraudMeNot uses. |
| Web Bot Auth (Signature-Agent) | Not documented; check for a Signature-Agent header if present. |
| robots.txt behavior | Ahrefs says AhrefsBot obeys robots.txt and Crawl-Delay; crawl-delay cannot apply during JavaScript rendering. |
Copy-Paste robots.txt & Cloudflare WAF Rules for AhrefsBot
robots.txt — allow public pages, keep private routes off-limits
User-agent: AhrefsBot
Allow: /
Disallow: /login/
Disallow: /account/
Disallow: /checkout/robots.txt — block completely
User-agent: AhrefsBot
Disallow: /Cloudflare WAF custom rule expression
(http.user_agent contains "AhrefsBot")Matches the User-Agent token, so it covers both real and spoofed traffic. Action: Block or Managed Challenge.
Capabilities, Data Usage & Autonomous Action Risk
| Trains AI on page content? | Not stated. Ahrefs says the crawl powers its SEO toolset and the Yep search engine. |
|---|---|
| Executes JavaScript? | Not documented |
| Can submit forms or attempt logins? | Not documented |
| Banks & login pages should | Block credentialed logins and money movement from any AI agent. |
| E-commerce checkouts should | Allow product page reads if you want; require human confirmation for checkout. |
Context: In October 2026, Axios reported that major U.S. banks were detecting unvetted third-party AI agents inside customer accounts, and CNBC reported that JPMorgan blocks outside AI agents from logging in with customers' credentials while allowing read-only data sharing through open banking providers.
Seen AhrefsBot? Quick Closed-Choice Tally
Choose from preset options only. No free-text comments, so nobody can post names, profanity, or unverified accusations.
Common Questions About AhrefsBot
What is AhrefsBot used for?
Ahrefs says it powers the database behind its marketing and SEO tools and the Yep search engine.
How do I slow AhrefsBot down?
Add a Crawl-Delay value in seconds to a "User-agent: AhrefsBot" group in robots.txt. Ahrefs says it follows this for HTML pages, though rendering can cause short bursts of asset requests.
How do I verify AhrefsBot?
Check the IP against Ahrefs' published ranges, or confirm the reverse DNS ends in ahrefs.com or ahrefs.net.
Could This Be a Scam? What to Do
- Anyone can copy the AhrefsBot User-Agent, and FraudMeNot found no official IP list for it, so the name alone proves nothing about who sent a request.
- Never give an AI agent your logins, bank access, or one-time codes unless you set it up yourself.
- If a call, text or email asking for money or access uses the AhrefsBot or Ahrefs name, stop and report it.
Guide: how to check whether a crawler visit is really who it claims to be.
More from Ahrefs
- AhrefsSiteAudit — SEO site-audit crawler
- AddSearchBot — AI Search Crawlers
- adidxbot — Advertising quality crawler
- AgentDataBot — AI Data Providers
- AgentTimes — Data Scraper from RSS Feeds
- AGI Agent — AI Agents
Related on FraudMeNot
- Verified security vulnerabilities, explained
- Report a scam, or find where to report it in your state
- Browse the AI Agent & Bot Directory
Sources (checked Nov. 16, 2026): ai.robots.txt (MIT), crawler-user-agents (MIT), Axios (Oct. 22, 2026), CNBC (Oct. 13, 2026), Ahrefs: Ahrefs Bots. FraudMeNot is not affiliated with Ahrefs. Product and company names are trademarks of their owners and are used here only to identify the bots. General information, not security advice.