I happened to load one of my datasets with dtype = str, that solved the issue.
df = pd.read_excel(file_path, dtype = str)