As @kmdreko mentioned, it worked by converting to euler and back, only keeping the y axis.

let (_, y, _) = camera_transform.rotation.to_euler(EulerRot::XYZ);
vel = Quat::from_euler(EulerRot::XYZ, 0., y, 0.) * vel * 10.;

Doing it without the euler conversion mostly alleviates the problem, but a slight slowing still happens.