In web scraping, you are doing a reverse process to get the data. That means you don't follow the requests from the beginning, you must follow them from the endpoint.

Remember, you are working against a back-end developer trying to prevent you from scraping the data.

How to do that?

• Follow the results (data) you aim to get from the scraping.
• Search for the results in the traffic, get the request responsible for loading it, and move backward step-by-step to understand what happened to get the data.

for example:

The link you are trying to scrap is loading the results in POST request not GET.

• Simulate the POST request and start from there.

Also don't forget to check the cookies, to see if it's mandatory to get the results or optional.