I examined the target website you're scraping, and since the data isn't loaded via a visible API, your current method is necessary. However, to improve efficiency and speed, consider using an asynchronous approach. Here’s how:
Use Scrapy – It's optimized for fast and scalable scraping.
Process in Batches – Instead of scraping pages one by one, fetch 100 at a time to reduce overhead.