I had to add

from PyInstaller.utils.hooks import collect_delvewheel_libs_directory
datas, binaries = collect_delvewheel_libs_directory("pandas")

to the hook-pandas.py file before it finally worked.