Overview

The leading Network of Pet and Hobby Supply Stores is a multi-divisional Internet company focused on the development of premium domains in the pet, equine, outdoor, and eco-friendly spaces.

The company’s platform and infrastructure offer online experiences that are at the pinnacle of their respective verticals. As just one example, the company is the largest online retailer of equine products worldwide.

The customers enjoy being able to simultaneously conduct eCommerce, get information, and be educated and entertained. The Chief Information Officer, oversees retail/direct selling, operations and merchandising.

Challenges

The company operates about fifteen active premium domains and, in total, has about 3,000 domains parked in some state of readiness. There are 50 or 60 SEO sites that help drive traffic to the company’s eCommerce sites.

Generic-sounding domains a magnet for unwanted traffic

“We have numerous dot com sites” says the company’s CIO. “Such generic-sounding domains lend themselves to a type of traffic that isn’t always desirable. It consumes bandwidth, taxes our infrastructure and distorts our metrics.

“What drove us to the Imperva solution is that we kept getting hit with unwanted traffic,” continued the CIO. “For example, we have a generic domain. It’s only three letters and easily recognizable, giving us some domain authority in relation to SEO ranking. But in being so generic, it also lends itself to spam, bots, and other lowlife traffic. The goal behind some of the bots remains a complete mystery. But then I don’t invest a lot of my time analyzing obtuse purposes that drive those.”

Serving alternate content not effective against advanced web scrapers

“I came from another well-known commence store, where we created a separate website and API from the primary domain—a place for that type of traffic to crawl. Eight or ten years ago that was a brilliant sideshow solution. But the scraper crowd has gotten a lot smarter and faster since then.”

Competitive data mining and price scraping bots killing response times for legitimate customers

“Most of the bots targeting our websites were competitive crawlers looking for full-load pricing information. Today there are all these crawlers that are smart enough to search, perform an ‘Add To Cart’ action, and scrape for price. All of those search operations represent a higher computational hit—it’s not simple HTML scraping. Such bots tax our backend servers and can really impact site performance,” says the CIO.

“I might be concurrently running 20 different promotions for 14 different sites. Pricing is dynamic between channel and email offers. The bots have become smart and fast enough these days to respond to the complexity of Internet marketing. They can hit, run through their scripts extremely quickly, and pull down our systems. If given free reign, they would be running a lot of queries against multiple scenarios for many different sites.”

The CIO says their infrastructure was getting overburdened with such unwanted, continual search requests and ultimately killing response times for legitimate customers. Having all of those bots carting items—a computationally intensive operation—put a lot of pressure on internal servers and databases. “In the eCommerce world, it’s a given that if customers can’t get to our sites, then there are no sales happening. Any such abrupt halt in the revenue stream—even a temporary one—presents a massive business problem,” he explained.

Tedious, manual bad bot investigations too little, too late

“Before Imperva I didn’t have the ability to measure if we were being hit by a bot or not,” he says. “We would get CPU spikes or experience slight slowdowns. ‘Is that a bot hitting us now?’ We’d start to look for patterns in the web logs, trying to ID offending IPs. Then we would go to the network firewall and try to block those addresses. It wasn’t pretty, but it was the only means we had at our disposal at the time.”

“In the end we were all left just scratching our heads. Digging into it ourselves was too labor-intensive and wasn’t our area of expertise. And by the time we would have figured what was happening, a problem instance would likely have come and gone.

The CIO says an explosion of bot activity was one of these things that might happen to them five times a year. “If you look at manhours and all the people involved, I could easily blow through 100 to 150 hours, all the while chasing ghosts.”

“And about three years ago we also got hit with a DDoS attack; that put us down for about a day and a half. I didn’t have any great way to stay ahead of it at the time.”

Lost sales. Competitors undercutting pricing in real-time to rank higher in search engine price sorts

“Everyone’s after a price competitive edge. Amazon has put a lot of pressure on every reseller—not just small or midsize, but every reseller. They change their prices numerous times throughout the day. If we drop our price to $1.99, an hour later it could be $1.98 at another site. Then they appear higher in search engine price sorts. Boom—we just lost a sale. Shopping loyalty is a challenge and keeping customers satisfied is difficult, if you just rely on price,” explains the CIO.

Results

Imperva Bot Management had just become available when he first went looking for a tool to tackle these problems, but already Imperva had a great reputation. “They were very responsive to our needs and willing to prove their worth through a proof of concept.”

“Putting a filter ahead of the junk traffic targeting our domain and servers made a whole lot of sense to us. Screening out that stuff is a plus, but we really sought to put an end to the competitive data mining and price scraping that undermines our competitive advantage.”

PoC demonstrated how bad bot problem could be solved in minutes or hours, versus days or weeks

“The PoC gave me an opportunity to take hard measurements and get the required internal approval from our board of directors,” says the CIO. “The entire enterprise had been down for a day and a half from the DDoS attack, so we could easily measure that in terms of lost sales. Once you fully grasp that there’s malicious intent on the Internet, then not incorporating this type of protection is irresponsible. Imperva is like having an insurance policy—it’s an easy sell. It lets my small team deal with situations within a few minutes or hours, versus days or weeks.”

Imperva Bot Management perfect complement to Verizon Edgecast

Imperva and Verizon filter website traffic to provide upfront protection to secure the eCommerce company’s business before the traffic reaches the companies origin server. Serving only good bot and human traffic allows for predictable changes to infrastructure capacity. Other than a SQL server, the company runs a cloud-based infrastructure. “These days I can scale horizontally by configuring however many web servers or app servers I want in the cloud. I can cycle them up in 15 minutes,” he said.

Eliminated bad bot traffic across entire network of eCommerce sites

“Before deploying Imperva Bot Management, I estimate that 20% of our traffic was bots, but it varied tremendously. One of our sites was almost 80% bots, while some of the premium domains were around 10%. Now we don’t have to deal them at all,” says the CIO.

“We’re saving about 10% in cloud infrastructure costs, as well as extending the life of internal servers by not overburdening them as they were, Imperva,” explains the CIO.

Imperva saves hundreds of hours per quarter, and makes everyone’s lives so much easier

“Our infrastructure team is small, so it’s important to engage with services that intelligently augment the team. I really like the reporting capabilities the Imperva Portal offers. It’s so intuitive that we don’t have to hire additional people to support all that it offers us. Being so utilitarian, Imperva Bot Management has made our lives so much easier.”

Clean analytics yield a better testing environment for ecommerce funnel optimization

“And the mere fact that there are legitimate customers visiting our site—versus bad bots—should yield a better testing environment and better analytical results. The marketing and metrics teams are just now coming up to speed with these additional benefits that Imperva offers, but I know they’re going to like what they see once they do, he adds.

Imperva Click Fraud Report provides evidence for discounts and credits with ad networks

“One report that especially interests me is the click fraud report in relation to pay-per-click advertisers. Previously, bots were clicking the ads and we had to pay for that bogus activity. Imperva providing that data really opened my eyes regarding all that it can do for us. Marketing was able to use those reports to turn off bot-ridden ad networks.

Tuning up for eCommerce holiday season

The CIO says they are probably using 85 or 90 percent of Imperva Bot Management capabilities. They plan to tune the solution before the busy holiday shopping season to get even more out of it. “Every website in the world has a bot problem,” he explained. “I heartily recommend Imperva Bot Management to solve the problem.