Sorry for the late answer.

It happens because results of even very basic operations such as matrix multiplication depend on input matrices size and when you are using KV-caching you change the matrix shape (for instance, in attention block). And the models were not trained with accordance to rounding errors caused by using different matrix sizes.

This issue has been discussed in huggingface github repo: https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535

Using KV caches, assisted generation, left-padding, and batching will change the logits. This happens in most, if not all models at all precisions, but it is almost imperceptible in FP32. With 16 bits, the difference becomes non-negligible. The model was not trained with KV caches or left-padding, so this is introducing a distribution shift -- it’s part of the cost of using a lower precision and other related optimizations. The effect is more visible when do_sample=True, as greedy decoding (do_sample=False) often selects the same token despite the differences.