The Roadmap to Reverse Engineering Mobile APIs for Data Extraction



A step by step guide with Charles Proxy and Android Emulator


This article about reverse-engineering Mobile API is written by Fabien Vauchelles, the Anti-Ban Expert at Wiremind – a leading revenue management solutions provider for the transportation, supply chain, and event sectors. With over a decade of experience in web scraping, Fabien’s passion for code and technology is unmatched. He is the mastermind behind Scrapoxy – a cloud-based proxy rotation tool – and is now working on the highly anticipated version 4.

When we try to scrape a site and struggle to retrieve the data, we often forget that there is also a mobile app. According to Brazilian researcher Tiago Bianchi, about 59% of internet traffic is mobile. So, why not take advantage of this? And most of the time, mobile app APIs are less protected than websites.

% of mobile traffic worldwide from 2015 to 2022
The Roadmap to Reverse Engineering Mobile APIs for Data Extraction 9

In this article, we will focus on Android app analysis. We will use the Android Studio IDE, which includes an emulator. We will connect Charles proxy, a software specialized in HTTP and HTTPS protocol analysis. It is extremely useful for designing or analyzing web and especially mobile applications. It even offers a root certificate to bypass SSL Pinning. Charles is an alternative to Fiddler, which Pierluigi presented in the first lab article.

Traffic Interception schema with Charles Proxy
The Roadmap to Reverse Engineering Mobile APIs for Data Extraction 10

Our environment is Ubuntu 22.04.

Part A: Setup of Charles

Note: We will use the Charles free trial, limited in usage time, which is more than enough for our needs.

Step 1 : Get Charles Proxy

Add the package from sources:

$ wget -q -O - https://www.charlesproxy.com/packages/apt/PublicKey | sudo apt-key add - 
$ sudo sh -c 'echo deb https://www.charlesproxy.com/packages/apt/ charles-proxy main > /etc/apt/sources.list.d/charles.list' 
$ sudo apt-get update 
$ sudo apt-get install charles-proxy 

Step 2: Enable SSL Proxying

Start Charles and open the menu Proxy > SSL Proxying Settings:

Charles Menu to Enable SSL Proxying
The Roadmap to Reverse Engineering Mobile APIs for Data Extraction 11

Click on Enable SSL Proxying and add a location *:*

Step 3: Download Charles certificate

Open the menu Help > SSL Proxying > Save Charles Root Certificate…, and save the certificate on your disk.

This post is sponsored by Smartproxy, the premium proxy and web scraping infrastructure focused on the best price, ease of use, and performance.

Smartproxy
Smartproxy

In this case, for all The Web Scraping Club Readers, using the discount code WEBSCRAPINGCLUB10 you can save 10% OFF for every purchase.

Step 4: Rename the certificate

The certificate cannot be used as is as a system certificate. We need to rename the file to match the format <hash>.0.

Let’s compute the hash:

$ openssl x509 -inform PEM -subject_hash_old -in charles-ssl-proxying-certificate.pem | head -1

Our hash is 4fe145fd. We need to rename the file :

$ mv charles-ssl-proxying-certificate.pem 4fe145fd.0

Let’s keep this certificate handy, we’ll use it again later.

Step 5 : Find your local IP

Open the menu Help > Local IP address:

Charles listing all local IP addresses
The Roadmap to Reverse Engineering Mobile APIs for Data Extraction 12

And remember your IP address !

For me, I always use my internal docker address because it never changes (here: 172.17.0.1)

Part B: Setup of an Android image

We will install Android Studio and create an image.

The full article is available only to paying users of the newsletter.
You can read this and other The Lab paid articles after subscribing


Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.



Liked the article? Subscribe for free to The Web Scraping Club to receive twice a week a new one in your inbox.