I tried many different things. The thing that solved it for me was including the OpenAPIVersion spec in UseSwagger():

    app.UseSwagger(options =>
    {
        options.OpenApiVersion = Microsoft.OpenApi.OpenApiSpecVersion.OpenApi2_0;
    });