You're dealing with an I/O-bound task since most of the time is spent waiting on the network, not doing CPU work. Starting a new Chrome for every URL is super heavy and burns through memory fast.

Switch to asyncio with Playwright so you can keep one browser open and load new tabs inside it. It's way more efficient. Use a semaphore or thread pool to limit how many tabs run at once, batch your URLs in chunks like 10k, and save results as you go. Also set up rotating proxies early so you don’t get blocked.