You can now use Portable Mode (https://code.visualstudio.com/docs/editor/portable

Essentially, download Zip version, unpack and in the main VSCode folder that you just extracted, create folder called data.

That is it, you now have fully portable version, all in one folder. Enjoy! :)