In ADF dataflow, I can see the expected output in the data preview section. However, when I run the pipeline, the final file is saved in Azure Blob Storage. I see that duplicate records for some reason. Some records are 14 times duplicate, some 10 times, some 7 times etc. I tried to tweak the partition settings as well, but to no avail.

For the issue you are facing I tried to find a work around. You can try following workflow to find. I have created one dataflow. In this source side I have added duplicate record.

I have performed one aggregate condition to check the count of duplicate records.

Then post that I have added conditional split. I have sent the output of duplicate records to blob storage container. In the output you can check Alice is a duplicate record. output of distinct I have sent to blob storage different container in the data preview you can see only distinct records are visible.