I have the exact same problem. Invalidating caches is a temp. solution. I have no idea why it's so slow. Also restarting the dart analysis server also works for a short while.
Reasons:
Low length (1):
No code block (0.5):
Me too answer (2.5): I have the exact same problem