In the current version (8.1.0) this can be achieved by using:
services.AddTransient<IApiControllerFilter, NoControllerFilter>();
See https://github.com/dotnet/aspnet-api-versioning/issues/1029 for details.