Solved this ! Just set Project Properties -> Linker -> Optimization -> References -> Yes (/OPT:REF)
This prevents the explicit dependency on oledlg.dll even when not being used.