A solution for Scalar is to empty the server list:

app.MapScalarApiReference(options =>
{
    options.Servers = [];
});

Credit to: https://github.com/dotnet/aspnetcore/issues/57332#issuecomment-2479286855

I am uncertain how to do this for Swagger with OpenAPI.