The directory my playwright tests were in did not have permissions to use rmvdir, which the html reporter uses at the end of a run to clear previous run results. Changing the directory solved the issue.