The problem was with my validating webhook where for Delete call I was taking req.Object.Raw instead of req.OldObject.Raw.