Here's an idea. Not sure if it is optimal but it should be correct.

1. sort each set.
2. pop the smallest item from each set into a minheap
3. track the max - min of elements in the heap
4. repeat:
   1. pop min from heap
   2. replace it the next smallest item from the same set (pop min from set again)
   3. (if said set is empty, stop. you've reached the optimal answer)

of course, you'd insert tuples of (value, parent_set) into the heap so you know which value came from which set once they're popped from the heap and needs to be replaced