I've been running into similar issues with several packages using an M1 Mac. So far I've found that uninstalling the package and then reinstalling it has worked:

#Uninstall the package pip3 uninstall your_package -y

#Reinstall it using the following command: pip3 install your_package --no-cache-dir --verbose --force-reinstall

Hope this works for you.