Hardware Retailer Nails Aggressive Price Scrapers with Netacea Bot Management

Category: Web Scraping


Reduction in traffic by blocking bots

Article Contents

    The Challenge

    The client is a US based online retailer operating dozens of ecommerce websites in the hardware industry. They also provide a suite of ecommerce services to other businesses within the home improvement sector.

    The client was being inundated by web scraper bots, which were accounting for as much as 75% of their web traffic. Although some web scrapers are acceptable, such as search engine indexing, others are either too aggressive or were crawling the websites with malicious intent.

    For example, consumers usually research home improvement purchases online to find the lowest possible price. Retailers use scraper bots to automatically undercut prices of their competitors and gain an advantage, damaging their rival retailers’ profit margins.

    Also, manufacturers frequently use minimum advertised price (MAP) agreements to prevent retailers selling their items below an agreed price floor. Breaking this agreement can lead to the manufacturer imposing restrictions on the seller. Competitors use pricing intelligence scraper bots to check whether retailers are adhering to MAP agreements, and report violations to the manufacturer, again to gain a competitive advantage.

    Performance, Infrastructure and Forecasting Issues

    Bot activity on our client’s websites had become extremely aggressive, especially in the wake of the global pandemic. With a variety of bots scraping the platform for different purposes at any given time, plus the day-to-day and sale traffic, the business was increasingly frustrated by performance degradation and outages. Customer experience was being affected negatively, leading to complaints and loss of customers to competitors.

    The prevalence of these bots also made it challenging to forecast traffic levels. The business found that their projections had gone from being predictable and consistent to quadrupling above anticipated levels due to bot traffic. The result of this was budgets for third parties being missed by a significant margin. The business also did not want to provision additional infrastructure just to serve bots that were potentially designed to harm their business.

    The business had previously attempted to block this kind of traffic using another bot management solution but found that most of the bots bypassed this layer of defense. They turned to Netacea’s advanced AI-powered bot management solution for assistance.

    The Solution

    Netacea was quickly put in place with an agentless implementation covering the entire estate of retail websites, removing the need to install and maintain individual JavaScript code on every site.

    The client first wanted to get a better insight into what these bots were doing and how much of this traffic should be blocked. Netacea’s Intent Analytics™ engine categorizes the intent behind every request made across their platform to determine whether it is non-human, and if so, whether it is part of a malicious attack.

    This is done using a multilayered approach, combining data from past attacks across Netacea’s user base with advanced machine learning algorithms to detect anomalous behavior. The output is also analyzed by Netacea’s team of bot experts.

    Combining these methods, attacks are quickly mitigated, either by passing suspected bad actors to a CAPTCHA challenge or by blocking them outright.

    Protect your websites, mobile apps and APIs from the threats posed by bots such as scrapers, scalpers, carding, credential stuffing and other automated attacks.

    The Outcome

    Netacea keeps bad bots at bay, regularly detecting attempted scraping efforts and blocking them from accessing the client’s websites with a high level of accuracy. Since implementing Netacea Bot Management, the business has reduced overall traffic by up to 50% by preventing scraper bots from crawling their sites.

    Attack Overview:

    • 810,961 malicious requests mitigated by Netacea over the course of three days
    • 100% “CAPTCHA incomplete” rate meaning extremely accurate mitigation
    • Potential outages avoided and infrastructure costs saved

    In just one attack attempt, scrapers attempted to flood one of the client’s websites with over 810,000 requests over the course of three days. Without mitigation, the peaks of this activity could have easily caused outages and serving these requests would have had significant financial and operational implications.

    Netacea’s Intent Analytics™ engine rapidly evaluated the behavior of each request, blocking known bad actors immediately and passing suspected bad bots to CAPTCHA for verification.

    None of the mitigated traffic completed CAPTCHA, confirming that they were indeed bots and that Netacea’s suggestions were extremely accurate.

    Block Bots Effortlessly with Netacea

    Book a demo and see how Netacea autonomously prevents sophisticated automated attacks.

    Related Case Studies

    US American Football cover art photo
    Case Study

    “The Big Game” Streamed Seamlessly to Millions Thanks to Netacea

    Netacea protected a major streaming service from outages during a major livestreaming event, mitigating huge credential stuffing attacks.
    Case Study

    Netacea Keeps an Online Pharmacy Safe from Scraping Attacks

    Aggressive scalper bots were threatening the availability of a major online pharmacy at peak times. Find out how Netacea protects them against malicious automation.
    Case Study

    Netacea Detects 11x More Bots Than Previous Bot Solution for Luxury Shoe Retailer

    Learn how Netacea helped a retailer of luxury shoe brands spot 11 times more bad bots than their previous solution, resulting in a 73% reduction in web traffic.

    Block Bots Effortlessly with Netacea

    Demo Netacea and see how our bot protection software autonomously prevents the most sophisticated and dynamic automated attacks across websites, apps and APIs.
    • Agentless, self managing spots up to 33x more threats
    • Automated, trusted defensive AI. Real-time detection and response
    • Invisible to attackers. Operates at the edge, deters persistent threats
    Book a Demo