I found the problem. There is not one, but TWO different ways to disable gradient computation and ComfyUI used the other one. Optimization works if with torch.torch.inference_mode(False): is prepended. I found that out thanks to https://github.com/comfyanonymous/ComfyUI/issues/2946