According to AWS Docs for AWS::ApiGatewayV2::Integration, for Websockets we need to always add IntegrationMethod: POST
AWS::ApiGatewayV2::Integration
IntegrationMethod: POST
I was missing that from my CloudFormation code