If anyone else is having this issue then it may also help to check what data types you have included in your df. Apparently primitive data types seem to be more likely to work correctly with drop_duplicates(). For example I had a df with PosixPaths that were all identical, but drop_duplicates() would not remove any of them until I changed the PosixPaths to strings.