I recommend you to use DrissionPage python library for web scraping and automation projects.

DrissionPage is a python-based web automation tool. It can control the browser, send and receive data packets, and combine the two into one. It can take into account the convenience of browser automation and the high efficiency of requests. It is powerful and has numerous user-friendly designs and convenient functions built in. Its syntax is concise and elegant, with little code and is friendly to novices.

Please reference the following url: https://drissionpage.cn/dp40docs/