Well, sometimes some websites have too many restrictions that make the scraping difficult. I would always suggest taking the help from a web scraping service provider when it comes to large scale scraping requirements. There are certain legal issues associated as well, when it comes to scraping. So, ideally, PromptCloud can help with them. Check them out: https://www.promptcloud.com/web-scraping-services/