I think in that case, Prometheus will need to scrape each node through it's exposed metrics endpoint. Because as you mentioned, if each node saves its own metrics locally in memory, then each time Prometheus calls the "/metrics" endpoint through the load balancer, it will get metric data from a random node based on the loadbalancer routing.

So, you will need Prometheus to scrape each node separately, which can be configured through prometheus.yaml file. For dynamic number of nodes behind the load balancer, you can rely on service discovery, which is also compatible with Prometheus.

Having Redis as a metrics collector for multiple instances is not recommended, as Redis acts as a single point of failure and can become a performance bottleneck.

If that's your case that you need to go with the push model where each node pushes its metrics,then you can check Prometheus push gateway.