The Lab #21: Navigating Anti-Bot Challenges with Artificial Intelligence

June is going to end soon and so is the AI month, in collaboration with Nimble. We talked about the future of AI and web scraping, how models depend on web data, and the changing landscape for the industry. Today we’ll be much more practical, using our classic testing methodology to see if the AI included in the Nimble browser could defeat the AI used in the most common anti-bot solutions.

AI vs AI: is this the anti-bot endgame?

Today, artificial intelligence is everywhere, and anti-bot solutions are not exempt from it.

This is the month of AI on The Web Scraping Club, in collaboration with Nimble.

During this period, we’ll learn more about how AI can be used for web scraping and its state of the art, I’ll try some cool products, and try to imagine how the future will look for the industry and, at the end of the month, a great surprise for all the readers.

Nimble Logo

And we could not have a better partner for this journey: AI applied to web scraping is the core expertise of the company and I’m sure we will learn something new by the end of the month.

If you open their website, you’ll read sentences like

Cloudflare’s Machine Learning trains on a curated subset of hundreds of billions of requests per day to create a reliable bot score for every request.


DataDome processes, in real time, 3 trillion data signals from client-side and server-side detection every day. Every single request to your mobile apps, websites, and APIs is analyzed and subjected to AI/machine learning models to determine in less than 2 milliseconds whether or not access should be granted to block threats from the first request.

For years, anti-bot solutions are using the billions of data points they receive daily, to train AI models to detect anomalies in fingerprints or user behavior.

At the same time, AI can also be used to generate reliable fingerprints and mimic human behavior, so it’s becoming a war between AI Models.

https%3A%2F%2Fsubstack post df97 4497 bb1a

In the past, we’ve already tested the “web unblocker” solutions in our “Hands On” series and today we’ll use the Nimble browser, a headful and automated browser specifically studied for web scraping, to check if it could defeat the most common anti-bot solutions.

The full article is available only to paying users of the newsletter.
You can read this and other The Lab paid articles after subscribing

Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.

Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.