Recently had this same issue in 2022.3.45f1, workaround was to do this after setting orthographicSize:

cam.orthographic = false;
cam.orthographic = true;