line-number-mode displays numbers in mode line so this is not applicable. You probably mean display-line-number-mode and it that case it should just work since the face used for the line numbers inherits from default.

In general, you can change arbitrary faces by adding them to auto-dim-other-buffers-affected-faces variable. For line numbers you’d need to add entry for line-number face.