Setting "Include Package JSON Auto Imports" to "On" instead of "auto" worked for me.
Not exactly sure why it improves performance, so if anyone knows, I would love an explanation.
Source: https://github.com/microsoft/TypeScript/issues/58709#issuecomment-2153332198