Do one single recursive scan.
Track deleted bytes instead of recalculating “after” size.
If speed is critical, consider using robocopy instead of Remove-Item.
robocopy
Remove-Item