The Lab #2: Data Extraction with DataDome & XSRF Tokens – A Complete Guide



A real-world use case of a simple scraper that does not get blocked by Datadome

What is Datadome?

Datadome Bot protection is one of the key players in the anti-bot software industry.

As stated on their website, the solution is used on several important websites such as FootLocker, Rakuten, and even Reddit, as you can see from the picture below.

1d55055e 156f 4caa a946 6bd165c479aa 1672x561 1
The Lab #2: Data Extraction with DataDome & XSRF Tokens – A Complete Guide 9

So, sooner or later, in your life as a web scraper, you’ll surely face one website protected with this technology.

These days I needed to update a scraper that eluded Datadome so it’s a good time for writing the process that allowed me to scrape the data from this website.

Are you looking for a Birkin?

In case you know what a Birkin is, you probably understood that the website in question is Hermes.com. For the others, a Birkin is one of the most iconic bags crafted by the Maison Hermes and it costs like a supercar (and no, it’s not sold online anyway).

7257b244 38ec 49df b77b 2f981be7fda7 1399x282
The Lab #2: Data Extraction with DataDome & XSRF Tokens – A Complete Guide 10

From a quick analysis of the network tab of the browser, we can see that by browsing the products in every category, we call an internal API that shows the product details we need.

Let’s start with the basic stuff

Let’s start with our standard Scrapy spider and see if we can get inside the website, after some make-up to our DEFAULT_REQUEST_HEADERS property.

65d04863 8c3e 42ef a6cb 68538fbd3e5f 1320x279
The Lab #2: Data Extraction with DataDome & XSRF Tokens – A Complete Guide 11

And soon we can see Datadome at work. We got redirected and locked out of the website, while on the browser the redirect leads to the Home page of the website.

dbc49741 e0da 4a16 8cb5 1c5fdbebf6ab 1081x526
The Lab #2: Data Extraction with DataDome & XSRF Tokens – A Complete Guide 12

A different path.

Hermes does not have an official App but I can use the same procedure explained in the first post of THE LAB to see how the website behaves when accessed by mobile.

The full article is available only to paying users of the newsletter.
You can read this and other The Lab paid articles after subscribing


Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.



Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.