The original cause was ConfigurationBuilder reloadOnChange functionality. The last boolean on this method indicates that reloadOnChange is active.

var baseBuilder = new ConfigurationBuilder().AddJsonFile(path, false, true).Build();

Even though the configuration builder gets cleaned up by garbage collection, the reloadOnChange had created a file handle that never got removed by garbage colleciton.