You can set .buttonStyle(.plain) on the button. This will won't alter your label anymore.
.buttonStyle(.plain)