I had the same issue and managed to delete rg.exe easily by running Windows in safe mode. After that, you're allowed to install VS Code again.
rg.exe