If the dtype becoming float is not a concern, then np.ma might be useful for working with this:

(np.ma.masked_invalid(df['a']) == np.ma.masked_invalid(df['b'])).astype(float).filled(np.nan)

This masks nan in the comparison, then replaces masked values back with nan.