Check if data annotations is ok. Then try to increase parameters of your ViT in config (num layers for ex.), maybe some experiments with path size will help.

Also it is very important to use pertained model. You should load some pretrained weights if you didn't.