So I appear to have figured out a solution, but it is somewhat of a hack:
One can keep track of the running VQ loss as another state variable of the RNN. Then, with return_states=True, one can access the final VQ loss via one of the return values of the RNN.
To get this to work, one has to use tf.fill to fill an entire tensor that in particular covers the batch size with the respective vq loss values. Finally, using the mentioned state output of the RNN, tf.reduce_mean will do the trick followed by an add_loss call.