All the answers before is telling you to rewrite the commit history, and change the branch name back and forth.
However, there is a much easy solution for this, You WON'T need to rewrite anything in your repo.
Just go to Account Settings of your GitHub account, and block the account you want.
You can unblock the account later.
At the moment you will still see the contributor showing on the list, but the number of contributors will be reduced by one.
After a few hours or a day, you will see that the contributor is no longer showing in the list.
Side-effect: The account you block will be disappeared from all the repo owned by you.