You can use the vite-plugin-inspect plugin to inspect which files vite is spending time processing. Then, let's disuss where the problem lies.
If not have benchmarks, there is no point in discussing performance optimization.