Turns out there were only some configurations missing.

So the general steps I did:

To start the agent:

eval "$(ssh-agent -s)"

Add the key:

ssh-add ~/.ssh/id_rsa

Create a config File:

git config --global core.sshCommand "C:/Program Files/Git/bin/ssh-agent.exe -s"

Set the Default SSH Socket:

git config --global core.sshCommand "ssh -i ~/.ssh/id_rsa"

This fixed my issues.