This issue mainly results from how class labels are encoded and loaded during training. Ensure that the labels are encoded and loaded the same way as you did during training so that the indexing is not messed up, that is if the classes are sorted during training make sure they are also sorted during inference. I hope this helps.