Based on a previous answer of @sonle I was able to work around this. Except that instead of using the .offset(x: 5) to align the toggle to the end, I used .scaleEffect(0.6, anchor: .trailing)