It turns out this was due to having both Python and Python Test Explorer for Visual Studio Code extensions enabled, both of them have test functionality. Disabling the latter fixed the issue.