Try adding
.buttonStyle(.plain)
to your bookmark button.
Its definitely very obscure and unintuitive but it somehow does the trick.