Based on this: https://github.com/dotnet/aspnetcore/issues/28547 and https://github.com/brave/brave-browser/issues/21364 it looks like Brave thinks closing the file browser is closing the browser window, so it stops debugging.

The solution for me was to stop the browser from automatically opening when debugging by setting "launchBrowser": false in launchSettings.json.