For me, this is what finally fixed it. I set the setting remote.SSH.remoteServerListenOnSocket to true:

{
  //...
  "remote.SSH.remoteServerListenOnSocket": true
}

then killed the VS Code server on the remote and retried my connection.