While the solutions by @BenzyNeez work, you can also fix this simply by changing from .paging to .viewAligned(limitBehavior: .always):
.paging
.viewAligned(limitBehavior: .always)
.scrollTargetBehavior(.viewAligned(limitBehavior: .always))