Giving a 2025 update on this question.

You need to pass the format string into pd.to_datetime, otherwise it pulls in dateutil and slowly parses each element individually (which may not be consistent). %p handles the AM/PM in the string.

pd.to_datetime(df['time'], format='%I:%M:%S %p')

To extract a clean time value you can then extract the .dt.time from it.