AI Bots & Web Crawlers Directory

Know exactly which AI crawlers, search engine bots, and automated agents reach your site: who runs them, what they do with your content, and how to allow or block each one.

All categories: Accessibility, Advertising, Agent, AI (No Training), AI Assistant, AI Crawler, AI Training, Analytics, E-commerce, Feed Fetcher, Monitoring, Preview, Search Engine Crawler, SEO, Social Media, User Initiated, Verification, Webhook

223 bots

AI Bots

AdagioBot

Adagiobot is a web crawler that analyzes websites for advertising demand optimization, helping publishers maximize revenue through real-time bidding analysis and performance insights.
Link

AdIdxBot

AdIdxBot is the crawler used by Bing Ads for quality control of ads and their destination websites. It has multiple user agent variants including desktop, iPhone, and Windows Phone versions.
Link

Adsense

The AdSense crawler visits participating sites in order to provide them with relevant ads.
Link

adsnaver

Naver's ad crawler that periodically visits registered ad landing pages to collect on-page content for effective ad matching and ranking. It ignores robots.txt for URLs registered in the ad system.
Link

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.
Link

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.
Link

AI2Bot

AI2Bot is operated by the Allen Institute for Artificial Intelligence (Ai2) to crawl the web for content to train open-source AI models.
Link

aiHitBot

aiHitBot collects and maintains historical information about companies.
Link

Algolia

The Algolia Crawler extracts content from your site and makes it searchable.
Link

Amazon AdBot

Amazon AdBot is a crawler used by different advertising services at Amazon to determine a website's content in order to provide relevant and appropriate advertising.
Link

Amazon Bedrock Bot

Amazon Bedrock Bot fetches web pages that customers add as data sources to Amazon Bedrock knowledge bases, so Bedrock-powered assistants can answer with that content.
Link

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.
Link

Amazon Product Discovery

Amazon's web crawler used to collect publicly available product details from Amazon Selling Partner websites to help improve the accuracy and completeness of product information on Amazon.
Link

Amazon Q

Amazon Q Business is a generative artificial intelligence (generative AI)-powered assistant that you can tailor to your business needs.
Link

Amazon Route 53 Health Check Service

Amazon Route 53 Health Check Service
Link

Amazon Seller Initiated Listing

Amazon's web crawler that helps sellers succeed by giving them the option to provide a URL to a website and create high-quality product pages in Amazon's store.
Link

Amazonbot

Amazonbot is Amazon's web crawler used to improve our services, such as enabling Alexa to more accurately answer questions for customers.
Link

Additional Content