As @MikeM commented, setting android:baselineAligned="false" on the <MaterialButtonToggleGroup> solved the issue.
<MaterialButtonToggleGroup>