Recently i had a same situation where i wanted to keep the route configuration within the appsettings.json and clusters config outside of appsettings.json so that i can dynamic update it.

I would suggest you to look into IProxyConfigProvider interface...using that you can do a bunch of work like dynamically updating your clusters address and more