I assume that you use the dataloader. Why don't you use the Duplicate detection feature ? It detects duplicates and continues to process the remaining records being inserted when detected.