Add this to your .devcontainer.json file:

"remoteEnv": {
    "SSH_AUTH_SOCK": ""
}