Botnet owners use them for large-scale malicious activity, commonly Distributed Denial of Service (DDoS) attacks. Botnets can also be used for any other malicious bot activity, such as spam bots or social bots (described below), albeit on a much larger scale.<\/p>\n
Types of Bots<\/h2>\n
There are many types of bots active on the Internet, both legitimate and malicious. Below are several common examples.<\/p>\n
Spider Bots<\/h3>\n
Spider bots, also known as web spiders or crawlers, browse the web by following hyperlinks, with the objective of retrieving and indexing web content. Spiders download HTML and other resources, such as CSS, JavaScript, and images, and use them to process site content.<\/p>\n
If you have a large number of web pages, you can place a robots.txt file in the root of your web server, and provide instructions to bots, specifying which parts of your site they can crawl, and how frequently.<\/p>\n
Scraper Bots<\/h3>\n
Scrapers are bots that read data from websites with the objective of saving them offline and enabling their reuse. This may take the form of scraping the entire content of web pages or scraping web content to obtain specific data points, such as names and prices of products on eCommerce sites.<\/p>\n
Web scraping<\/a> is a gray area -in some cases, scraping is legitimate and may be permitted by website owners. In other cases, bot operators may be violating website terms of use, or worse\u2014leveraging scraping to steal sensitive or copyrighted content.<\/p>\n A spambot is an Internet application designed to gather email addresses for spam mailing lists. A spam bot can gather emails from websites, social media websites, businesses and organizations, leveraging the distinctive format of email addresses.<\/p>\n After attackers have amassed a large list of email addresses, they can use them not only to send spam email, but also for other nefarious purposes:<\/p>\n Apart from the direct damage caused to end-users and organizations affected by spam campaigns, spam bots can also choke server bandwidth and increase costs for Internet Service Providers (ISPs).<\/p>\n Bots are operated on social media networks, and used to automatically generate messages, advocate ideas, act as a follower of users, and as fake accounts to gain followers themselves. It is estimated that 9-15% of Twitter accounts are social bots.<\/p>\n Social bots can be used to infiltrate groups of people and used to propagate specific ideas. Since there is no strict regulation governing their activity, social bots play a major role in online public opinion.<\/p>\n Social bots can create fake accounts (although this is becoming more difficult as social networks become more sophisticated), amplify the bot operator\u2019s message, and generate fake followers\/likes. It is difficult to identify and mitigate social bots, because they can exhibit very similar behavior to that of real users.<\/p>\n Download bots are automated programs that can be used to automatically download software or mobile apps. They can be used to influence download statistics, for example to gain more downloads on popular app stores and help new apps get to the top of the charts. They can also be used to attack download sites, creating fake downloads as part of an application-layer Denial of Service (DoS) attack.<\/p>\n Ticketing Bots are an automated way to purchase tickets to popular events, with the aim of reselling those tickets for a profit. This activity is illegal in many countries, and even if not prohibited by law, it is an annoyance to event organizers, ticket sellers and consumers.<\/p>\n Ticketing bots tend to be very sophisticated, emulating the same behaviors as human ticket buyers. In many ticketing domains, the proportion of tickets purchased by automated bots ranges between 40-95%.<\/p>\n Following are a few parameters you can use in a manual check of your web analytics, to detect bot traffic hitting a website:<\/p>\n All of the above are only rough indicators of bot activity. Be aware that sophisticated malicious bots can generate a realistic, user-like signature in your web analytics. It is advisable to use a dedicated bot management<\/a> solution that provides full visibility of bot traffic.<\/p>\n There are a few simple measures you can take to block at least some bots and reduce your exposure to bad bots:<\/p>\n Bot technology has evolved over the past decade. Originally, bots were a script hitting a website to retrieve data or perform actions. These scripts would not accept cookies and did not parse JavaScript, making them very easy to detect.<\/p>\n Over time bots got more sophisticated, accepting cookies and parsing JavaScript, but they could still be detected quite easily because they used dynamic website elements less than human users.<\/p>\n The next evolution was the use of headless browsers like PhantomJS\u2014these can process website content in its entirety. Although these browsers are more sophisticated than basic bots, headless browsers still cannot perform all actions that real users can.<\/p>\n The most advanced types of bots are based on the Chrome browser and are almost indistinguishable from real users. These bots even simulate human activity such as clicking on-page elements.<\/p>\n As bots evolved, so did mitigation techniques. There are currently three technical approaches to detecting and mitigating bad bots:<\/p>\n By combining the three approaches, you can overcome evasive bots of all types, and successfully separate them from human traffic. You can use these approaches independently or you can rely on bot mitigation services to perform techniques for you.<\/p>\n Bot mitigation services are automated tools that use the above methods to identify bots. These services can be used to monitor API traffic and detect if it is legitimate machine traffic or bad bots \u201cmilking\u201d your API.<\/p>\n Advanced bot mitigation services use rate limiting for each requesting client or machine, instead of an entire IP, allowing it to limit crawling from bad bots. Once a bot is identified, these services can propagate the information across the network, to ensure the same bot cannot access your site or API again.<\/p>\n See how Imperva Bot Management can help you with automated threats.<\/p>\n Imperva\u2019s bot management solution uses all three approaches covered above\u2014static, challenge-based and behavior-based\u2014to investigate each visitor on your site, whether human or not, and match it with a behavioral ID. It can effectively protect against malicious bots while ensuring that legitimate bots and human users have uninterrupted access to your site.<\/p>\n Imperva Bot Management: Detection methods and response options<\/p><\/div>\n In addition to helping mitigate bad bot activity, Imperva provides multi-layered protection to make sure websites and applications are available, easily accessible and safe. The Imperva application security solution includes:<\/p>\nSpam Bots<\/h3>\n
\n
Social Media Bots<\/h3>\n
Download Bots<\/h3>\n
Ticketing Bots<\/h3>\n
How To Detect Bot Traffic in Web Analytics<\/h2>\n
\n
\nSuspicious hits from single IPs\u2014a big number of hits from a single IP. Humans typically request a few pages and not others, whereas bots will often request all pages.<\/li>\nHow to Stop Bot Traffic: Basic Mitigation Measures<\/h2>\n
\n
How Do Bots Evade Detection?<\/h2>\n
Advanced Bot Mitigation Techniques<\/h2>\n
\n
Imperva Bot Management<\/h2>\n
![\"Imperva](\"https:\/\/www.imperva.com\/learn\/wp-content\/uploads\/sites\/13\/2019\/12\/what-are-bots-detection.jpg\")
\n