A Closer Look at the Most Active Good Bots

The fifth annual Imperva Incapsula Bot Traffic Report discussed the latest trends in bot traffic, including an analysis of good and bad bot activities.

Bot activity as a whole increased over the past year, attributable mainly to the uptick in good bot traffic. Zooming in, we identified the most active good bots that generated over 84 percent of all good bot traffic.

There were 35 bots that comprised the vast majority of good bot traffic in 2016. They can be grouped into four categories: search engine bots, commercial crawlers, feed fetchers, and monitoring bots.

Search Engine Bots

These are bots used by search engines to crawl websites, check links, retrieve contend and update their indicies.

• Googlebot – A search engine bot used by Google, the most widely used search engine on the internet.
• Yandex bot – A search bot used by Yandex, the largest search engine in Russia.
• MSN/Bingbot – A search bot used by Microsoft to index websites for Bing search engine. It replaced the MSN bot in 2010.
• Yahoo! Slurp bot – One of the search bots (along with Bingbot) used by Yahoo search engine.
• MJ12Bot – A search bot used to index websites for Majestic-12 , a UK based search engine and SEO backlink checker.
• Baiduspider – A search bot used by Baidu, the leading Chinese search engine.
• Mail.Ru bot – A search engine bot used by the Mail.Ru group, Russia’s leading email service.
• Haosou 360 spider – A search bot used by Qihoo 360, China’s second most used search engine.
• Sogou spider – A search bot used by Sogou, one of China’s leading search engines.
• Google AdsBot – A search bot that inspects the quality of landing pages, which are then ranked accordingly in Google searches.

Commercial Crawlers

These are bots used by commercial enterprises to crawl websites and retrieve information.

• AhrefsBot – A crawler bot operated by Ahrefs, a marketing and SEO tool primarily used as a backlink checker.
• Proximic bot – A crawler bot used by Proximic, a platform for matching ad campaigns to website content.
• SEMrushBot – A crawler bot operated by SEMrush, a platform that provides SEO data, including keyword research and backlinks.
• Dotmic DotBot – A crawler bot that identifies e-commerce websites, product names, pricing data and descriptions.
• Pinterest bot – A crawler bot used by Pinterest, a social media application that allows users to share and manage photo collections.
• YisouSpider – A crawler bot operated by Chinese search engine Yisou.
• XoviBot – A crawler bot utilized by Xovi GmbH, a marketing solution that provides SEO, search analytics and link management services.
• ADmantX bot – A crawler bot operated by ADmantX, a data analysis provider that helps marketers develop effective online campaigns.
• Grapeshot bot – A crawler bot used by Grapeshot, a keyword technology company, which analyzes the content of webpages for automated page classification.
• Alexa bot – A crawler bot employed by Alexa, a subsidiary of Amazon that provides data and analytics on website traffic.

Feed Fetchers

These are bots programmed to retrieve data that can be displayed on websites.

• Facebook Mobile app – A feed fetcher that retrieves website information to be viewed in Facebook’s in-app browser.
• Android Framework bot – A feed fetcher used by Android Runtime environment to retrieve content for mobile apps.
• CFNetwork bot – A feed fetcher used by the CFNetwork framework in iPhone apps.
• .NET WebClient – A feed fetcher associated with the .NET framework, a software framework that runs primarily on Microsoft Windows.
• Facebook external hit – A feed fetcher used by Facebook to retrieve details, such as thumbnail images or video tags, associated with shared links.
• Windows Phone WebClient – A feed fetcher used by Windows Phone webclient to fetch website information for viewing in mobile apps.
• Adobe AIR – A feed fetcher used by Adobe AIR runtime environment to retrieve content for native apps running on desktop and mobile platforms.
• Google Feedfetcher – A feed fetcher used by Google to retrieve RSS and Atom feeds for display on website homepages.
• Twitter bot – A feed fetcher that retrieves details associated with external links displayed on Twitter.
• FeedBurner – A feed fetcher used by publishers to manage custom RSS feeds.

Monitoring Bots

These are bots used to monitor various performance metrics on websites.

• WordPress (pingback) – A monitoring bot that allows WordPress users to notify bloggers that they have linked to their site.
• Paessler Monitor – A monitoring bot used by Paessler, a web based platform that monitors networks for system outages.
• UptimeBot – A monitoring bot used by Uptime, a website performance monitoring service.
• Site24X7 tools – A monitoring bot used by Site24x7, a website performance monitoring service.
• Keynote – A monitoring bot operated by Keynote, a provider of cloud-based mobile app monitoring solutions.

For more information on good bot and bad bot activity trends, and their implications for your website, check out the full 2016 Bot Traffic Report.