I found a better solution: I built a custom RNN where I store the sample vq losses in a Tensorarray and return this to the outer model, where the array is summed and the resulting loss is added to the model.