After trying some random methods of UIButton, I found that it is enough to add
button.tinitAdjustmentMode = .normal