A solution for Scalar is to empty the server list:
app.MapScalarApiReference(options =>
{
options.Servers = [];
});
Credit to: https://github.com/dotnet/aspnetcore/issues/57332#issuecomment-2479286855
I am uncertain how to do this for Swagger with OpenAPI.