Try using single_utterance=true or manually half-close the stream when the API sends an END_OF_SINGLE_UTTERANCE response.

If this will not work, the issue needs to be investigated further. Please open a new issue on the issue tracker, describing your problem.