First I generated a fine-grained token and tried multiple times, it didn't work. Then I generated a personal access token and clicked repo, and it worked. Sometimes you just need to use the PAT instead of the fine-grained ones.
repo