What helped me:

Clear / Delete the Extension-Cache:

Close VS Code

Delete the content inside the workspaceStorage.

~/Library/Application Support/Code/User/workspaceStorage

For Windows it is:

C:\Users\username\AppData\Roaming\Code\User\workspaceStorage

• Start VSCode

And it should be faster / normal again

At least that was my final step and its smoothly running again 🙏