AI Agent & Automated Bot Directory

AI Agent & Bot Directory

Look up 211 known models, crawlers, agents, and bots from the open-source ai.robots.txt registry and operator documentation: who runs them, whether a crawler respects robots.txt (76 yes · 10 no · 125 unclear), and how to tell a real crawler from a spoofed User-Agent. A model or browser agent is not given a crawler identity it does not publish.

Compare Bots

Full AI Agent Reports (Preview)

All 211 Known AI Agents & Crawlers

Top categories: AI Assistants (30), AI Data Providers (20), AI Agents (18), AI Data Scrapers (18), AI Search Crawlers (14), AI Coding Agents (9).

Agent tokenOperatorPurposeRespects robots.txt
AddSearchBotUnknown operatorAI Search CrawlersUnclear
adidxbotMicrosoft (Bing)Advertising quality crawlerYes
AgentDataBotUnknown operatorAI Data ProvidersUnclear
AgentTimesThe Agent TimesData Scraper from RSS FeedsUnclear
AGI AgentAGI, IncAI AgentsUnclear
AhrefsBotAhrefsSEO crawlerYes
AhrefsSiteAuditAhrefsSEO site-audit crawlerYes
AI2BotAi2Content is used to train open language modelsYes
AI2Bot-DeepResearchEvalAi2, a non-profit AI research instituteAI AssistantsUnclear
Ai2Bot-DolmaAi2Content is used to train open language modelsYes
aiHitBotaiHitA massive, artificial intelligence/machine learning, automatYes
AIWebIndexLyrenthAI Search CrawlersYes
amazon-kendraAmazonCollects data for AI natural language searchYes
amazon-QBusinessUnknown operatorAI AssistantsUnclear
AmazonbotAmazonService improvement and enabling answers for Alexa usersYes
AmazonBuyForMeAmazonAI AgentsUnclear
Amzn-SearchBotUnknown operatorAI Search CrawlersUnclear
Amzn-UserAmazon, used for fetching web content to answer user queries through Alexa and other Amazon AI servicesAI AssistantsUnclear
AndibotAndiSearch engine using generative AI, AI Search AssistantUnclear
AnomuraDireqtCollects data for AI searchYes
anthropic-aiAnthropicScrapes data to train Anthropic's AI productsUnclear
ApifyBotUnknown operatorAI Data ProvidersUnclear
ApifyWebsiteContentCrawlerUnknown operatorAI Data ProvidersUnclear
ApplebotUnknown operatorAI Search CrawlersYes
Applebot-ExtendedApplePowers features in Siri, Spotlight, Safari, Apple IntelligenYes
Aranet-SearchBotUnknown operatorUndocumented AI AgentsUnclear
atlassian-botAtlassianAI search, assistants and agentsYes
AwarioAwarioAI Data ScrapersUnclear
AzureAI-SearchBotUnknown operatorAI Search CrawlersUnclear
bedrockbotAmazonData scraping for custom AI applicationsYes
bigsur.aiBig Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi…AI AssistantsUnclear
bingbotMicrosoft (Bing)Search engine crawlerYes
BingPreviewMicrosoft (Bing)Page snapshot crawlerYes
BingVideoPreviewMicrosoft (Bing)Video preview crawlerYes
BixelBotUnknown operatorAI Data ProvidersUnclear
Bravebothttps://safe.search.brave.com/help/brave-search-crawlerAI Data ProvidersYes
BrightbotUnknown operatorAI Data ProvidersUnclear
Brightbot 1.0https://brightdata.com/brightbotLLM/AI trainingUnclear
BuddyBotBuddyBotLearningAI Learning CompanionUnclear
BytespiderByteDanceLLM trainingNo
CartAI Agentic CommerceCartAIAI AgentsUnclear
CCBotCommon Crawl FoundationProvides open crawl dataset, used for many purposes, includiYes
Channel3BotUnknown operatorAI Search CrawlersUnclear
ChatGLM-SpiderUnknown operatorAI Data ScrapersUnclear
ChatGPT AgentOpenAIAI AgentsYes
ChatGPT-UserOpenAIAI AssistantsYes
Claude-CodeUnknown operatorAI Coding AgentsUnclear
Claude-SearchBotAnthropicClaude-SearchBot navigates the web to improve search result Yes
Claude-UserAnthropicAI AssistantsYes
Claude-WebAnthropicUndocumented AI AgentsUnclear
ClaudeBotAnthropicScrapes data to train Anthropic's AI productsYes
CledaraBotCledaraAI AgentsUnclear
Cloudflare-AutoRAGCloudflareCollects data for AI searchYes
CloudflareBrowserRenderingCrawlerCloudflare that returns rendered website content for research, monitoring, and AI data workflowsAI Data ProvidersUnclear
CloudVertexBotUnknown operatorAI Data ScrapersUnclear
CodeUnknown operatorAI Coding AgentsUnclear
CodexUnknown operatorAI Coding AgentsUnclear
cohere-aiCohereRetrieves data to provide responses to user-initiated promptUnclear
cohere-training-data-crawlerCohere to download training data for its LLMs (Large Language Models) that power its enterprise AI productsAI Data ScrapersUnclear
CotoyogiROISAI LLM ScraperYes
CragCrawlerCragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping servicesAI Data ProvidersUnclear
Crawl4AIUnknown operatorUndocumented AI AgentsUnclear
CrawlspaceCrawlspaceAI Data ProvidersYes
CursorUnknown operatorAI Coding AgentsUnclear
Datenbank CrawlerDatenbankAI Data ScrapersUnclear
DeepSeekBotDeepSeekTraining language models and improving AI productsNo
DevinDevin AIAI Coding AgentsYes
DiffbotDiffbotAI Data ProvidersUnclear
Diffbot-UserDiffbotAI AssistantsYes
DotUnknown operatorAI AgentsUnclear
DoubaoBotByteDanceAI crawler for ByteDance's Doubao AI assistantUnclear
DuckAssistBotUnknown operatorAI AssistantsYes
DuckDuckBotDuckDuckGoSearch engine crawlerYes
Echobot BotEchoboxAI Data ScrapersUnclear
EchoboxBotEchoboxData collection to support AI-powered productsUnclear
ElevenlabsBotElevenLabsAI AssistantsYes
ElevenlabsCrawlerElevenLabsAI Data ScraperYes
ERNIEBotBaiduCollects public web content for Baidu's ERNIE large languageUnclear
ExaBotUnknown operatorAI Data ProvidersUnclear
ExaSearchBotExaAI Search CrawlersUnclear
ExaUserUnknown operatorAI AssistantsUnclear
FacebookBotMeta/FacebookTraining language modelsYes
facebookexternalhitMeta/FacebookOstensibly only for sharing, but likely used as an AI crawleNo
Factset_spyderbotFactsetAI model trainingUnclear
FirecrawlAgentFirecrawl that extracts web content and converts it into structured data for use in LLM and AI applicationsAI Data ProvidersYes
FriendlyCrawlerUnknown operatorWe are using the data from the crawler to build datasets forYes
GeistHaus-PageFetcherGeistHaus, a company developing AI systems for therapy and psychological assessmentAI AssistantsUnclear
Gemini-Deep-ResearchUnknown operatorAI AssistantsUnclear
Gemma 4 12BGoogle DeepMindAI modelUnclear
Google-AgentUnknown operatorAI AgentsYes
Google-CloudVertexBotGoogleBuild and manage AI models for businesses employing Vertex AYes
Google-ExtendedGoogleLLM trainingYes
Google-FirebaseGoogleUsed as part of AI apps developed by users of Google's FirebUnclear
Google-Gemini-CLIUnknown operatorAI Coding AgentsUnclear
Google-InspectionToolGoogleSearch testing fetcherYes
Google-GeminiNotebookUnknown operatorAI AssistantsUnclear
GoogleAgent-MarinerGoogleAI AgentsUnclear
GoogleAgent-URLContextGoogle that retrieves web content on behalf of Gemini API usersAI AssistantsUnclear
GooglebotGoogleSearch engine crawlerYes
Googlebot-ImageGoogleSearch image crawlerYes
Googlebot-VideoGoogleSearch video crawlerYes
GoogleOtherGoogleScrapes dataYes
GoogleOther-ImageGoogleScrapes dataYes
GoogleOther-VideoGoogleScrapes dataYes
GPTBotOpenAIScrapes data to train OpenAI's productsYes
Grok BotUnknown operatorAI AgentsUnclear
GrokAppAndroidxAIAI Search CrawlersUnclear
GrokBotxAIAI Search CrawlersUnclear
HarkBotUnknown operatorAI AgentsUnclear
HenkBotUnknown operatorAI Data ProvidersUnclear
HunyuanTencentHunyuan is a training crawler operated by TencentUnclear
iAskBotUnknown operatorUndocumented AI AgentsUnclear
iaskspideriAskUndocumented AI AgentsNo
ICC-CrawlerNICTScrapes data to train and support AI technologiesYes
ImagesiftBotImageSiftImageSiftBot is a web crawler that scrapes the internet for Yes
imageSpiderUnknown operatorAI Data ScrapersUnclear
img2datasetimg2datasetScrapes images for use in LLMsUnclear
InstinctUnknown operatorAI AgentsUnclear
ISSCyberRiskCrawlerISS-CorporateScrapes data to train machine learning modelsNo
kagi-fetcherKagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res…AI AssistantsUnclear
Kangaroo BotUnknown operatorAI Data ScrapersUnclear
KeenableBotKeenableAI Search CrawlersUnclear
Kimi-AgentMoonshot AI that browses websites while carrying out user-directed research and other tasksAI AgentsUnclear
Kimi-SearchBotMoonshot AIAI Search CrawlersYes
Kimi-UserMoonshot AI that fetches web content on behalf of users interacting with KimiAI AssistantsUnclear
KimiBotMoonshot AIAI Data ScrapersYes
KlaviyoAIBotKlaviyoAI AssistantsYes
KunatoCrawlerUnknown operatorUndocumented AI AgentsUnclear
laion-huggingface-processorUnknown operatorAI Data ScrapersUnclear
LAIONDownloaderLarge-scale Artificial Intelligence Open NetworkAI tools and models for machine learning researchNo
LCCUnknown operatorAI Data ScrapersUnclear
LightpandaGrok-adjacentAI Data ScrapersUnclear
LinerBotUnknown operatorAI AssistantsUnclear
Linguee BotLingueeAI powered translation serviceNo
LinkupBotUnknown operatorAI Search CrawlersUnclear
Manus-UserButterfly Effect, a company based in ChinaAI AgentsUnclear
Meta-ExternalAgentMetaUsed to train models and improve productsYes
Meta-ExternalFetcherUnknown operatorAI AssistantsNo
meta-webindexerMetaAI AssistantsUnclear
MicrosoftPreviewMicrosoftPage snapshot crawlerYes
MistralAI-IndexMistral AIIndexes web content for Mistral AI's search, used to answer Yes
MistralAI-TrainingMistral AIAI Data ScrapersYes
MistralAI-UserMistralAI AssistantsYes
MoonshotBotKimiMoonshotBot is a training crawlerUnclear
Mozilla-TabstackMozilla that performs programmatic, AI-driven interactions with web content through TabstackAI Data ProvidersYes
MuseUnknown operatorAI AgentsUnclear
MyCentralAIScraperBotUnknown operatorAI data scraperUnclear
NagetBotNaget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts)AI data scraperUnclear
nbotNBotAI Search CrawlersUnclear
netEstate Imprint CrawlernetEstateAI Data ScrapersUnclear
newsaiUnknown operatorAI data scraperUnclear
NotebookLMUnknown operatorAI AssistantsUnclear
NovaActUnknown operatorAI AgentsUnclear
OAI-AdsBotOpenAIValidates and targets ads on ChatGPTUnclear
OAI-SearchBotOpenAISearch result generationYes
omgiliWebz.ioData is soldYes
omgilibotWebz.ioData is soldYes
OpenAIOpenAIUnclear at this timeYes
opencodeUnknown operatorAI Coding AgentsUnclear
OperatorUnknown operatorAI AgentsUnclear
PanguBotthe Chinese company HuaweiAI Data ScrapersUnclear
PanscientPanscientData collection and analysis using machine learning and AIYes
panscient.comPanscientData collection and analysis using machine learning and AIYes
Payhawk Invoice Fetching AgentPayhawkAI AgentsUnclear
Perplexity-UserPerplexityAI AssistantsNo
PerplexityBotPerplexitySearch result generationYes
PetalBotHuaweiUsed to provide recommendations in Hauwei assistant and AI sYes
PhindBotphindAI AssistantsUnclear
Poggio-CitationsPoggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan…AI AssistantsUnclear
Poseidon Research CrawlerPoseidon ResearchAI research crawlerUnclear
qodercliQoder that helps developers build, debug, and modify software from the terminalAI Coding AgentsUnclear
QualifiedBotQualifiedAI AssistantsUnclear
Querit-SearchBotQuerit that indexes web content for their search API service, which is designed to provide real-time search results for larg…AI Data ProvidersUnclear
QueritBotQuerit, a company providing a search API for large language model integrationAI Data ProvidersUnclear
QuillBotQuillbotCompany offers AI detection, writing tools and other serviceUnclear
quillbot.comQuillbotCompany offers AI detection, writing tools and other serviceUnclear
QwenBotAlibabaCollects public web content for Alibaba's Qwen (Tongyi QianwUnclear
ReflectionbotReflectionUndocumented AI AgentsUnclear
SBIntuitionsBotSB IntuitionsUses data gathered in AI development and information analysiYes
ScrapyZyteScrapes data for a variety of uses including training AIUnclear
ScryBotScryUsed with AI agents to scrape data for answers to queriesUnclear
SemrushBot-OCOBSemrushCrawls your site for ContentShake AI toolYes
SemrushBot-SWASemrushChecks URLs on your site for SEO Writing AssistantYes
Shap-UserUnknown operatorAI AssistantsUnclear
ShapBotParallelAI Data ProvidersYes
Sidetrade indexer botSidetradeExtracts data for a variety of uses including training AIUnclear
SpiderUnknown operatorAI Data ScrapersUnclear
Storebot-GoogleGoogleShopping crawlerYes
TavilyBotUnknown operatorAI Data ProvidersUnclear
Terra CottaUnknown operatorAI Data ProvidersUnclear
TerraCottaCeramic AIAI Data ProvidersYes
ThinkbotThinkbotInsights on AI integration and automationNo
TikTokSpiderByteDanceLLM trainingUnclear
TimpibotTimpiScrapes data for use in training LLMsUnclear
TongyiBotAlibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answersAI AssistantsUnclear
TraeUnknown operatorAI Coding AgentsUnclear
TwinAgentTwin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa…AI AgentsUnclear
UseAIUnknown operatorAI AssistantsUnclear
VelenPublicWebCrawlerVelen CrawlerScrapes data for business data sets and machine learning modYes
WARDBotWEBSPARKAI Data ScrapersUnclear
Webzio-ExtendedUnknown operatorAI Data ScrapersUnclear
wpbotQuantumCloudLive chat support and lead generationUnclear
WRTNBotUnknown operatorUndocumented AI AgentsUnclear
xAI-GrokxAIxAI-Grok is an xAI crawler token associated with Grok contenUnclear
YaKMeltwaterAccording to the [Meltwater Consumer Intelligence page](httpUnclear
YandexAdditionalYandexScrapes/analyzes data for the YandexGPT LLMYes
YandexAdditionalBotYandexScrapes/analyzes data for the YandexGPT LLMYes
YiBot01.aiThis bot crawls the web to gather data and content for indexUnclear
YiyanBotBaidu that fetches web content for the yiyanAI AssistantsUnclear
YouBotYouScrapes data for search engine and LLMsYes
ZanistaBotUnknown operatorAI Search CrawlersUnclear

Sources (checked Nov. 16, 2026): ai.robots.txt registry, used under the MIT License (Copyright © the ai.robots.txt contributors); crawler-user-agents, used under the MIT License (Copyright © Martin Monperrus and contributors); operator-published IP feeds from OpenAI, Google, Apple and Perplexity. Entries with the same name in different letter case or with a /version suffix are merged. FraudMeNot is not affiliated with any operator listed.