As of January 2025, I had the same problem with very slow step-by-step Python debugging.
I found the culprit thanks to a comment by Jossy: the Watch panel.
Hiding the Watch panel and/or deleting all unnecessary watches solved the problem: now it is extremely responsive.