AI Agent & Bot Directory
Look up 211 known models, crawlers, agents, and bots from the open-source ai.robots.txt registry and operator documentation: who runs them, whether a crawler respects robots.txt (76 yes · 10 no · 125 unclear), and how to tell a real crawler from a spoofed User-Agent. A model or browser agent is not given a crawler identity it does not publish.
Compare Bots
Full AI Agent Reports (Preview)
- AddSearchBot — Unknown operator · Listed AI Agent
- adidxbot — Microsoft (Bing) · Listed AI Agent
- AgentDataBot — Unknown operator · Listed AI Agent
- AgentTimes — The Agent Times · AI Data Collection Bot
- AGI Agent — AGI, Inc (claimed, unverified) · Listed AI Agent
- AhrefsBot — Ahrefs · Listed AI Agent
- AhrefsSiteAudit — Ahrefs · Listed AI Agent
- AI2Bot — Ai2 · AI Data Collection Bot
- AI2Bot-DeepResearchEval — Ai2, a non-profit AI research institute · AI Agent (User-Triggered)
- Ai2Bot-Dolma — Ai2 (claimed, unverified) · AI Data Collection Bot
- aiHitBot — aiHit · Listed AI Agent
- AIWebIndex — Lyrenth · Listed AI Agent
- amazon-kendra — Amazon Web Services · Listed AI Agent
- amazon-QBusiness — Amazon Web Services · AI Agent (User-Triggered)
- Amazonbot — Amazon · Listed AI Agent
- AmazonBuyForMe — Amazon · Listed AI Agent
- Amzn-SearchBot — Amazon · Listed AI Agent
- Amzn-User — Amazon · AI Agent (User-Triggered)
- Andibot — Andi · AI Agent (User-Triggered)
- Anomura — Direqt · Listed AI Agent
- anthropic-ai — Anthropic · AI Data Collection Bot
- ApifyBot — Apify (claimed, unverified) · Listed AI Agent
- ApifyWebsiteContentCrawler — Apify (claimed, unverified) · Listed AI Agent
- Applebot — Apple · Listed AI Agent
- Applebot-Extended — Apple · Listed AI Agent
- Aranet-SearchBot — Unknown operator · Listed AI Agent
- atlassian-bot — Atlassian · AI Agent (User-Triggered)
- Awario — Awario · AI Data Collection Bot
- AzureAI-SearchBot — Microsoft (claimed, unverified) · Listed AI Agent
- bedrockbot — Amazon · AI Data Collection Bot
- bigsur.ai — Big Sur AI · AI Agent (User-Triggered)
- bingbot — Microsoft (Bing) · Listed AI Agent
- BingPreview — Microsoft (Bing) · Listed AI Agent
- BingVideoPreview — Microsoft (Bing) · Listed AI Agent
- BixelBot — Unknown operator · Listed AI Agent
- Bravebot — Brave · Listed AI Agent
- Brightbot — Bright Data · Listed AI Agent
- BuddyBot — BuddyBotLearning · Listed AI Agent
- Bytespider — ByteDance · High Caution — Aggressive Scraper
- CartAI Agentic Commerce — CartAI (claimed, unverified) · Listed AI Agent
- CCBot — Common Crawl Foundation · Listed AI Agent
- Channel3Bot — Unknown operator · Listed AI Agent
- ChatGLM-Spider — Unknown operator · AI Data Collection Bot
- ChatGPT Agent — OpenAI · Listed AI Agent
- ChatGPT-User — OpenAI · Verified AI Agent (User-Triggered)
- Claude-Code — Anthropic (claimed, unverified) · Listed AI Agent
- Claude-SearchBot — Anthropic · Listed AI Agent
- Claude-User — Anthropic · AI Agent (User-Triggered)
- Claude-Web — Anthropic · Listed AI Agent
- ClaudeBot — Anthropic · AI Data Collection Bot
- CledaraBot — Cledara (claimed, unverified) · Listed AI Agent
- Cloudflare-AutoRAG — Cloudflare · Listed AI Agent
- CloudflareBrowserRenderingCrawler — Cloudflare · Listed AI Agent
- Code — GitHub (claimed, unverified) · Listed AI Agent
- Codex — OpenAI (claimed, unverified) · Listed AI Agent
- cohere-training-data-crawler — Cohere (claimed, unverified) · AI Data Collection Bot
- Cotoyogi — ROIS (Research Organization of Information and Systems) · AI Data Collection Bot
- CragCrawler — CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services · Listed AI Agent
- Crawl4AI — Unknown operator · Listed AI Agent
- Crawlspace — Crawlspace · Listed AI Agent
- Cursor — Unknown operator · Listed AI Agent
- Datenbank Crawler — Datenbank · AI Data Collection Bot
- DeepSeekBot — DeepSeek · Caution — Ignores robots.txt
- Devin — Devin AI · Listed AI Agent
- Diffbot — Diffbot · Listed AI Agent
- Diffbot-User — Diffbot · AI Agent (User-Triggered)
- Dot — OpenAI (claimed, unverified) · Listed AI Agent
- DoubaoBot — ByteDance (claimed, unverified) · AI Agent (User-Triggered)
- DuckAssistBot — DuckDuckGo · AI Agent (User-Triggered)
- DuckDuckBot — DuckDuckGo · Listed AI Agent
- Echobot Bot — Echobox · AI Data Collection Bot
- EchoboxBot — Echobox · Listed AI Agent
- ElevenlabsBot — ElevenLabs · AI Agent (User-Triggered)
- ElevenlabsCrawler — ElevenLabs · AI Data Collection Bot
- ERNIEBot — Baidu (claimed, unverified) · Listed AI Agent
- ExaBot — Exa (claimed, unverified) · Listed AI Agent
- ExaSearchBot — Exa · Listed AI Agent
- ExaUser — Exa · AI Agent (User-Triggered)
- FacebookBot — Meta · AI Data Collection Bot
- facebookexternalhit — Meta · Caution — Ignores robots.txt
- Factset_spyderbot — Factset · AI Data Collection Bot
- FirecrawlAgent — Firecrawl · Listed AI Agent
- FriendlyCrawler — Unknown operator · Listed AI Agent
- GeistHaus-PageFetcher — GeistHaus, a company developing AI systems for therapy and psychological assessment · AI Agent (User-Triggered)
- Gemini-Deep-Research — Google (claimed, unverified) · AI Agent (User-Triggered)
- Gemma 4 12B — Google DeepMind (claimed, unverified) · Listed AI Agent
- Google-Agent — Google · Listed AI Agent
- Google-CloudVertexBot — Google · Listed AI Agent
- Google-Extended — Google · AI Data Collection Bot
- Google-Firebase — Google (claimed, unverified) · Listed AI Agent
- Google-Gemini-CLI — Google · Listed AI Agent
- Google-InspectionTool — Google · Listed AI Agent
- Google-GeminiNotebook — Google · AI Agent (User-Triggered)
- GoogleAgent-Mariner — Google · Listed AI Agent
- GoogleAgent-URLContext — Google (claimed, unverified) · AI Agent (User-Triggered)
- Googlebot — Google · Listed AI Agent
- Googlebot-Image — Google · Listed AI Agent
- Googlebot-Video — Google · Listed AI Agent
- GoogleOther — Google · AI Data Collection Bot
- GoogleOther-Image — Google · AI Data Collection Bot
- GoogleOther-Video — Google · AI Data Collection Bot
- GPTBot — OpenAI · Verified AI Crawler (Training)
- Grok Bot — xAI (claimed, unverified) · Listed AI Agent
- GrokAppAndroid — xAI (claimed, unverified) · Listed AI Agent
- GrokBot — xAI (claimed, unverified) · Listed AI Agent
- HarkBot — Hark (claimed, unverified) · Listed AI Agent
- HenkBot — Valyu · Listed AI Agent
- Hunyuan — Tencent (claimed, unverified) · AI Data Collection Bot
- iAskBot — Unknown operator · Listed AI Agent
- iaskspider — iAsk · Caution — Ignores robots.txt
- ICC-Crawler — NICT (National Institute of Information and Communications Technology, Japan) · AI Data Collection Bot
- ImagesiftBot — ImageSift (Hive) · AI Data Collection Bot
- imageSpider — Unknown operator · AI Data Collection Bot
- img2dataset — img2dataset (open-source tool) · AI Data Collection Bot
- Instinct — Unknown operator · Listed AI Agent
- ISSCyberRiskCrawler — ISS-Corporate · Caution — Ignores robots.txt
- kagi-fetcher — Kagi · AI Agent (User-Triggered)
- Kangaroo Bot — Unknown operator · AI Data Collection Bot
- KeenableBot — Keenable (claimed, unverified) · Listed AI Agent
- Kimi-Agent — Moonshot AI (Kimi) (claimed, unverified) · Listed AI Agent
- Kimi-SearchBot — Moonshot AI · Listed AI Agent
- Kimi-User — Moonshot AI (Kimi) · AI Agent (User-Triggered)
- KimiBot — Moonshot AI · AI Data Collection Bot
- KlaviyoAIBot — Klaviyo · AI Agent (User-Triggered)
- KunatoCrawler — Unknown operator · Listed AI Agent
- laion-huggingface-processor — Unknown operator · AI Data Collection Bot
- LAIONDownloader — Large-scale Artificial Intelligence Open Network · Caution — Ignores robots.txt
- LCC — Unknown operator · AI Data Collection Bot
- Lightpanda — Grok-adjacent · AI Data Collection Bot
- LinerBot — Liner · AI Agent (User-Triggered)
- Linguee Bot — Linguee · Caution — Ignores robots.txt
- LinkupBot — Linkup · Listed AI Agent
- Manus-User — Manus (claimed, unverified) · AI Agent (User-Triggered)
- Meta-ExternalAgent — Meta · AI Data Collection Bot
- Meta-ExternalFetcher — Meta · AI Agent (User-Triggered)
- meta-webindexer — Meta · AI Agent (User-Triggered)
- MicrosoftPreview — Microsoft · Listed AI Agent
- MistralAI-Index — Mistral AI · AI Data Collection Bot
- MistralAI-Training — Mistral AI · AI Data Collection Bot
- MistralAI-User — Mistral AI · AI Agent (User-Triggered)
- MoonshotBot — Kimi · AI Data Collection Bot
- Mozilla-Tabstack — Mozilla (Tabstack) · Listed AI Agent
- Muse — Meta (claimed, unverified) · Listed AI Agent
- MyCentralAIScraperBot — Unknown operator · AI Data Collection Bot
- NagetBot — Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) · AI Data Collection Bot
- nbot — NBot · Listed AI Agent
- netEstate Imprint Crawler — netEstate · AI Data Collection Bot
- newsai — Unknown operator · AI Data Collection Bot
- NovaAct — Amazon · Listed AI Agent
- OAI-AdsBot — OpenAI · Listed AI Agent
- OAI-SearchBot — OpenAI · Verified Search Crawler (ChatGPT search)
- omgili — Webz.io · Listed AI Agent
- omgilibot — Webz.io · Listed AI Agent
- opencode — OpenCode (open source) · Listed AI Agent
- Operator — OpenAI (claimed, unverified) · Listed AI Agent
- PanguBot — Huawei · AI Data Collection Bot
- panscient.com — Panscient · Listed AI Agent
- Payhawk Invoice Fetching Agent — Payhawk (claimed, unverified) · Listed AI Agent
- Perplexity-User — Perplexity · AI Agent (User-Triggered)
- PerplexityBot — Perplexity · Listed AI Agent
- PetalBot — Huawei · AI Agent (User-Triggered)
- PhindBot — phind · AI Agent (User-Triggered)
- Poggio-Citations — Poggio · AI Agent (User-Triggered)
- Poseidon Research Crawler — Poseidon Research · Listed AI Agent
- qodercli — Qoder that helps developers build, debug, and modify software from the terminal · Listed AI Agent
- QualifiedBot — Qualified · AI Agent (User-Triggered)
- Querit-SearchBot — Querit (claimed, unverified) · Listed AI Agent
- QueritBot — Querit (claimed, unverified) · Listed AI Agent
- QuillBot — Quillbot · Listed AI Agent
- quillbot.com — Quillbot · Listed AI Agent
- QwenBot — Alibaba (claimed, unverified) · Listed AI Agent
- Reflectionbot — Reflection · Listed AI Agent
- SBIntuitionsBot — SB Intuitions · Listed AI Agent
- Scrapy — Scrapy (open-source, maintained by Zyte) · AI Data Collection Bot
- ScryBot — Scry (claimed, unverified) · AI Data Collection Bot
- SemrushBot-OCOB — Semrush · Listed AI Agent
- SemrushBot-SWA — Semrush · AI Agent (User-Triggered)
- Shap-User — Parallel Web Systems · AI Agent (User-Triggered)
- ShapBot — Parallel · Listed AI Agent
- Sidetrade indexer bot — Sidetrade · AI Data Collection Bot
- Spider — Unknown operator · AI Data Collection Bot
- Storebot-Google — Google · Listed AI Agent
- TavilyBot — Tavily (claimed, unverified) · Listed AI Agent
- Terra Cotta — Ceramic (claimed, unverified) · Listed AI Agent
- TerraCotta — Ceramic AI · Listed AI Agent
- Thinkbot — Thinkbot (operator unconfirmed) · Caution — Ignores robots.txt
- TikTokSpider — ByteDance (claimed, unverified) · AI Data Collection Bot
- Timpibot — Timpi · AI Data Collection Bot
- TongyiBot — Alibaba (claimed, unverified) · AI Agent (User-Triggered)
- Trae — ByteDance (claimed, unverified) · Listed AI Agent
- TwinAgent — Twin · Listed AI Agent
- UseAI — Unknown operator · AI Agent (User-Triggered)
- VelenPublicWebCrawler — Velen · AI Data Collection Bot
- WARDBot — WEBSPARK (W.A.R.D.) · AI Data Collection Bot
- Webzio-Extended — Webz.io · AI Data Collection Bot
- wpbot — QuantumCloud · Listed AI Agent
- WRTNBot — Unknown operator · Listed AI Agent
- xAI-Grok — xAI (claimed, unverified) · Listed AI Agent
- YaK — Meltwater · Listed AI Agent
- YandexAdditional — Yandex · AI Data Collection Bot
- YiBot — 01.ai · AI Data Collection Bot
- YiyanBot — Baidu (claimed, unverified) · AI Agent (User-Triggered)
- YouBot — You.com · AI Data Collection Bot
- ZanistaBot — Unknown operator · Listed AI Agent
All 211 Known AI Agents & Crawlers
Top categories: AI Assistants (30), AI Data Providers (20), AI Agents (18), AI Data Scrapers (18), AI Search Crawlers (14), AI Coding Agents (9).
| Agent token | Operator | Purpose | Respects robots.txt |
|---|---|---|---|
| AddSearchBot | Unknown operator | AI Search Crawlers | Unclear |
| adidxbot | Microsoft (Bing) | Advertising quality crawler | Yes |
| AgentDataBot | Unknown operator | AI Data Providers | Unclear |
| AgentTimes | The Agent Times | Data Scraper from RSS Feeds | Unclear |
| AGI Agent | AGI, Inc | AI Agents | Unclear |
| AhrefsBot | Ahrefs | SEO crawler | Yes |
| AhrefsSiteAudit | Ahrefs | SEO site-audit crawler | Yes |
| AI2Bot | Ai2 | Content is used to train open language models | Yes |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute | AI Assistants | Unclear |
| Ai2Bot-Dolma | Ai2 | Content is used to train open language models | Yes |
| aiHitBot | aiHit | A massive, artificial intelligence/machine learning, automat | Yes |
| AIWebIndex | Lyrenth | AI Search Crawlers | Yes |
| amazon-kendra | Amazon | Collects data for AI natural language search | Yes |
| amazon-QBusiness | Unknown operator | AI Assistants | Unclear |
| Amazonbot | Amazon | Service improvement and enabling answers for Alexa users | Yes |
| AmazonBuyForMe | Amazon | AI Agents | Unclear |
| Amzn-SearchBot | Unknown operator | AI Search Crawlers | Unclear |
| Amzn-User | Amazon, used for fetching web content to answer user queries through Alexa and other Amazon AI services | AI Assistants | Unclear |
| Andibot | Andi | Search engine using generative AI, AI Search Assistant | Unclear |
| Anomura | Direqt | Collects data for AI search | Yes |
| anthropic-ai | Anthropic | Scrapes data to train Anthropic's AI products | Unclear |
| ApifyBot | Unknown operator | AI Data Providers | Unclear |
| ApifyWebsiteContentCrawler | Unknown operator | AI Data Providers | Unclear |
| Applebot | Unknown operator | AI Search Crawlers | Yes |
| Applebot-Extended | Apple | Powers features in Siri, Spotlight, Safari, Apple Intelligen | Yes |
| Aranet-SearchBot | Unknown operator | Undocumented AI Agents | Unclear |
| atlassian-bot | Atlassian | AI search, assistants and agents | Yes |
| Awario | Awario | AI Data Scrapers | Unclear |
| AzureAI-SearchBot | Unknown operator | AI Search Crawlers | Unclear |
| bedrockbot | Amazon | Data scraping for custom AI applications | Yes |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… | AI Assistants | Unclear |
| bingbot | Microsoft (Bing) | Search engine crawler | Yes |
| BingPreview | Microsoft (Bing) | Page snapshot crawler | Yes |
| BingVideoPreview | Microsoft (Bing) | Video preview crawler | Yes |
| BixelBot | Unknown operator | AI Data Providers | Unclear |
| Bravebot | https://safe.search.brave.com/help/brave-search-crawler | AI Data Providers | Yes |
| Brightbot | Unknown operator | AI Data Providers | Unclear |
Brightbot 1.0 | https://brightdata.com/brightbot | LLM/AI training | Unclear |
| BuddyBot | BuddyBotLearning | AI Learning Companion | Unclear |
| Bytespider | ByteDance | LLM training | No |
| CartAI Agentic Commerce | CartAI | AI Agents | Unclear |
| CCBot | Common Crawl Foundation | Provides open crawl dataset, used for many purposes, includi | Yes |
| Channel3Bot | Unknown operator | AI Search Crawlers | Unclear |
| ChatGLM-Spider | Unknown operator | AI Data Scrapers | Unclear |
| ChatGPT Agent | OpenAI | AI Agents | Yes |
| ChatGPT-User | OpenAI | AI Assistants | Yes |
| Claude-Code | Unknown operator | AI Coding Agents | Unclear |
| Claude-SearchBot | Anthropic | Claude-SearchBot navigates the web to improve search result | Yes |
| Claude-User | Anthropic | AI Assistants | Yes |
| Claude-Web | Anthropic | Undocumented AI Agents | Unclear |
| ClaudeBot | Anthropic | Scrapes data to train Anthropic's AI products | Yes |
| CledaraBot | Cledara | AI Agents | Unclear |
| Cloudflare-AutoRAG | Cloudflare | Collects data for AI search | Yes |
| CloudflareBrowserRenderingCrawler | Cloudflare that returns rendered website content for research, monitoring, and AI data workflows | AI Data Providers | Unclear |
CloudVertexBot | Unknown operator | AI Data Scrapers | Unclear |
| Code | Unknown operator | AI Coding Agents | Unclear |
| Codex | Unknown operator | AI Coding Agents | Unclear |
cohere-ai | Cohere | Retrieves data to provide responses to user-initiated prompt | Unclear |
| cohere-training-data-crawler | Cohere to download training data for its LLMs (Large Language Models) that power its enterprise AI products | AI Data Scrapers | Unclear |
| Cotoyogi | ROIS | AI LLM Scraper | Yes |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services | AI Data Providers | Unclear |
| Crawl4AI | Unknown operator | Undocumented AI Agents | Unclear |
| Crawlspace | Crawlspace | AI Data Providers | Yes |
| Cursor | Unknown operator | AI Coding Agents | Unclear |
| Datenbank Crawler | Datenbank | AI Data Scrapers | Unclear |
| DeepSeekBot | DeepSeek | Training language models and improving AI products | No |
| Devin | Devin AI | AI Coding Agents | Yes |
| Diffbot | Diffbot | AI Data Providers | Unclear |
| Diffbot-User | Diffbot | AI Assistants | Yes |
| Dot | Unknown operator | AI Agents | Unclear |
| DoubaoBot | ByteDance | AI crawler for ByteDance's Doubao AI assistant | Unclear |
| DuckAssistBot | Unknown operator | AI Assistants | Yes |
| DuckDuckBot | DuckDuckGo | Search engine crawler | Yes |
| Echobot Bot | Echobox | AI Data Scrapers | Unclear |
| EchoboxBot | Echobox | Data collection to support AI-powered products | Unclear |
| ElevenlabsBot | ElevenLabs | AI Assistants | Yes |
| ElevenlabsCrawler | ElevenLabs | AI Data Scraper | Yes |
| ERNIEBot | Baidu | Collects public web content for Baidu's ERNIE large language | Unclear |
| ExaBot | Unknown operator | AI Data Providers | Unclear |
| ExaSearchBot | Exa | AI Search Crawlers | Unclear |
| ExaUser | Unknown operator | AI Assistants | Unclear |
| FacebookBot | Meta/Facebook | Training language models | Yes |
| facebookexternalhit | Meta/Facebook | Ostensibly only for sharing, but likely used as an AI crawle | No |
| Factset_spyderbot | Factset | AI model training | Unclear |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications | AI Data Providers | Yes |
| FriendlyCrawler | Unknown operator | We are using the data from the crawler to build datasets for | Yes |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment | AI Assistants | Unclear |
| Gemini-Deep-Research | Unknown operator | AI Assistants | Unclear |
| Gemma 4 12B | Google DeepMind | AI model | Unclear |
| Google-Agent | Unknown operator | AI Agents | Yes |
| Google-CloudVertexBot | Build and manage AI models for businesses employing Vertex A | Yes | |
| Google-Extended | LLM training | Yes | |
| Google-Firebase | Used as part of AI apps developed by users of Google's Fireb | Unclear | |
| Google-Gemini-CLI | Unknown operator | AI Coding Agents | Unclear |
| Google-InspectionTool | Search testing fetcher | Yes | |
| Google-GeminiNotebook | Unknown operator | AI Assistants | Unclear |
| GoogleAgent-Mariner | AI Agents | Unclear | |
| GoogleAgent-URLContext | Google that retrieves web content on behalf of Gemini API users | AI Assistants | Unclear |
| Googlebot | Search engine crawler | Yes | |
| Googlebot-Image | Search image crawler | Yes | |
| Googlebot-Video | Search video crawler | Yes | |
| GoogleOther | Scrapes data | Yes | |
| GoogleOther-Image | Scrapes data | Yes | |
| GoogleOther-Video | Scrapes data | Yes | |
| GPTBot | OpenAI | Scrapes data to train OpenAI's products | Yes |
| Grok Bot | Unknown operator | AI Agents | Unclear |
| GrokAppAndroid | xAI | AI Search Crawlers | Unclear |
| GrokBot | xAI | AI Search Crawlers | Unclear |
| HarkBot | Unknown operator | AI Agents | Unclear |
| HenkBot | Unknown operator | AI Data Providers | Unclear |
| Hunyuan | Tencent | Hunyuan is a training crawler operated by Tencent | Unclear |
| iAskBot | Unknown operator | Undocumented AI Agents | Unclear |
| iaskspider | iAsk | Undocumented AI Agents | No |
| ICC-Crawler | NICT | Scrapes data to train and support AI technologies | Yes |
| ImagesiftBot | ImageSift | ImageSiftBot is a web crawler that scrapes the internet for | Yes |
| imageSpider | Unknown operator | AI Data Scrapers | Unclear |
| img2dataset | img2dataset | Scrapes images for use in LLMs | Unclear |
| Instinct | Unknown operator | AI Agents | Unclear |
| ISSCyberRiskCrawler | ISS-Corporate | Scrapes data to train machine learning models | No |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… | AI Assistants | Unclear |
| Kangaroo Bot | Unknown operator | AI Data Scrapers | Unclear |
| KeenableBot | Keenable | AI Search Crawlers | Unclear |
| Kimi-Agent | Moonshot AI that browses websites while carrying out user-directed research and other tasks | AI Agents | Unclear |
| Kimi-SearchBot | Moonshot AI | AI Search Crawlers | Yes |
| Kimi-User | Moonshot AI that fetches web content on behalf of users interacting with Kimi | AI Assistants | Unclear |
| KimiBot | Moonshot AI | AI Data Scrapers | Yes |
| KlaviyoAIBot | Klaviyo | AI Assistants | Yes |
| KunatoCrawler | Unknown operator | Undocumented AI Agents | Unclear |
| laion-huggingface-processor | Unknown operator | AI Data Scrapers | Unclear |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network | AI tools and models for machine learning research | No |
| LCC | Unknown operator | AI Data Scrapers | Unclear |
| Lightpanda | Grok-adjacent | AI Data Scrapers | Unclear |
| LinerBot | Unknown operator | AI Assistants | Unclear |
| Linguee Bot | Linguee | AI powered translation service | No |
| LinkupBot | Unknown operator | AI Search Crawlers | Unclear |
| Manus-User | Butterfly Effect, a company based in China | AI Agents | Unclear |
| Meta-ExternalAgent | Meta | Used to train models and improve products | Yes |
| Meta-ExternalFetcher | Unknown operator | AI Assistants | No |
| meta-webindexer | Meta | AI Assistants | Unclear |
| MicrosoftPreview | Microsoft | Page snapshot crawler | Yes |
| MistralAI-Index | Mistral AI | Indexes web content for Mistral AI's search, used to answer | Yes |
| MistralAI-Training | Mistral AI | AI Data Scrapers | Yes |
| MistralAI-User | Mistral | AI Assistants | Yes |
| MoonshotBot | Kimi | MoonshotBot is a training crawler | Unclear |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack | AI Data Providers | Yes |
| Muse | Unknown operator | AI Agents | Unclear |
| MyCentralAIScraperBot | Unknown operator | AI data scraper | Unclear |
| NagetBot | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | AI data scraper | Unclear |
| nbot | NBot | AI Search Crawlers | Unclear |
| netEstate Imprint Crawler | netEstate | AI Data Scrapers | Unclear |
| newsai | Unknown operator | AI data scraper | Unclear |
NotebookLM | Unknown operator | AI Assistants | Unclear |
| NovaAct | Unknown operator | AI Agents | Unclear |
| OAI-AdsBot | OpenAI | Validates and targets ads on ChatGPT | Unclear |
| OAI-SearchBot | OpenAI | Search result generation | Yes |
| omgili | Webz.io | Data is sold | Yes |
| omgilibot | Webz.io | Data is sold | Yes |
OpenAI | OpenAI | Unclear at this time | Yes |
| opencode | Unknown operator | AI Coding Agents | Unclear |
| Operator | Unknown operator | AI Agents | Unclear |
| PanguBot | the Chinese company Huawei | AI Data Scrapers | Unclear |
Panscient | Panscient | Data collection and analysis using machine learning and AI | Yes |
| panscient.com | Panscient | Data collection and analysis using machine learning and AI | Yes |
| Payhawk Invoice Fetching Agent | Payhawk | AI Agents | Unclear |
| Perplexity-User | Perplexity | AI Assistants | No |
| PerplexityBot | Perplexity | Search result generation | Yes |
| PetalBot | Huawei | Used to provide recommendations in Hauwei assistant and AI s | Yes |
| PhindBot | phind | AI Assistants | Unclear |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… | AI Assistants | Unclear |
| Poseidon Research Crawler | Poseidon Research | AI research crawler | Unclear |
| qodercli | Qoder that helps developers build, debug, and modify software from the terminal | AI Coding Agents | Unclear |
| QualifiedBot | Qualified | AI Assistants | Unclear |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… | AI Data Providers | Unclear |
| QueritBot | Querit, a company providing a search API for large language model integration | AI Data Providers | Unclear |
| QuillBot | Quillbot | Company offers AI detection, writing tools and other service | Unclear |
| quillbot.com | Quillbot | Company offers AI detection, writing tools and other service | Unclear |
| QwenBot | Alibaba | Collects public web content for Alibaba's Qwen (Tongyi Qianw | Unclear |
| Reflectionbot | Reflection | Undocumented AI Agents | Unclear |
| SBIntuitionsBot | SB Intuitions | Uses data gathered in AI development and information analysi | Yes |
| Scrapy | Zyte | Scrapes data for a variety of uses including training AI | Unclear |
| ScryBot | Scry | Used with AI agents to scrape data for answers to queries | Unclear |
| SemrushBot-OCOB | Semrush | Crawls your site for ContentShake AI tool | Yes |
| SemrushBot-SWA | Semrush | Checks URLs on your site for SEO Writing Assistant | Yes |
| Shap-User | Unknown operator | AI Assistants | Unclear |
| ShapBot | Parallel | AI Data Providers | Yes |
| Sidetrade indexer bot | Sidetrade | Extracts data for a variety of uses including training AI | Unclear |
| Spider | Unknown operator | AI Data Scrapers | Unclear |
| Storebot-Google | Shopping crawler | Yes | |
| TavilyBot | Unknown operator | AI Data Providers | Unclear |
| Terra Cotta | Unknown operator | AI Data Providers | Unclear |
| TerraCotta | Ceramic AI | AI Data Providers | Yes |
| Thinkbot | Thinkbot | Insights on AI integration and automation | No |
| TikTokSpider | ByteDance | LLM training | Unclear |
| Timpibot | Timpi | Scrapes data for use in training LLMs | Unclear |
| TongyiBot | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers | AI Assistants | Unclear |
| Trae | Unknown operator | AI Coding Agents | Unclear |
| TwinAgent | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… | AI Agents | Unclear |
| UseAI | Unknown operator | AI Assistants | Unclear |
| VelenPublicWebCrawler | Velen Crawler | Scrapes data for business data sets and machine learning mod | Yes |
| WARDBot | WEBSPARK | AI Data Scrapers | Unclear |
| Webzio-Extended | Unknown operator | AI Data Scrapers | Unclear |
| wpbot | QuantumCloud | Live chat support and lead generation | Unclear |
| WRTNBot | Unknown operator | Undocumented AI Agents | Unclear |
| xAI-Grok | xAI | xAI-Grok is an xAI crawler token associated with Grok conten | Unclear |
| YaK | Meltwater | According to the [Meltwater Consumer Intelligence page](http | Unclear |
| YandexAdditional | Yandex | Scrapes/analyzes data for the YandexGPT LLM | Yes |
YandexAdditionalBot | Yandex | Scrapes/analyzes data for the YandexGPT LLM | Yes |
| YiBot | 01.ai | This bot crawls the web to gather data and content for index | Unclear |
| YiyanBot | Baidu that fetches web content for the yiyan | AI Assistants | Unclear |
| YouBot | You | Scrapes data for search engine and LLMs | Yes |
| ZanistaBot | Unknown operator | AI Search Crawlers | Unclear |
Sources (checked Nov. 16, 2026): ai.robots.txt registry, used under the MIT License (Copyright © the ai.robots.txt contributors); crawler-user-agents, used under the MIT License (Copyright © Martin Monperrus and contributors); operator-published IP feeds from OpenAI, Google, Apple and Perplexity. Entries with the same name in different letter case or with a /version suffix are merged. FraudMeNot is not affiliated with any operator listed.