Deduplication worked as intended. However, since the table was partitioned by the event_date field, the original batch was split into smaller batches for each partition under the hood. Thus the hash that used by the deduplication mechanism was computed separately for each of these partitioned batches.