The best option i found was to use

<mat-slide-toggle [checked]="isChecked()" (mousedown)="toggle($event)" />

then in the function

toggle(event: any) {
  event.preventDefault();
}

this basically helps to capture the mouse event before any click event is done