On my end, the server was defaulting to IPv6, so I temporarily disabled it.

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

This solved the issue of getting stuck at: Downloading VS code server