I recommend you should use SMOTE for oversampling or under sampling or you shuffle your data before splitting and use Stratified K fold cross validation which helps to ensure each fold has the same proportion of classes