My issue was solved by this: https://discuss.huggingface.co/t/valueerror-target-size-torch-size-8-must-be-the-same-as-input-size-torch-size-8-8/12133/10
You need to keep the label as integers.