Assuming all your requests will come from the same address, you can add in your VirtualHost a ServerAlias parameter (you can define multiples for the same virtualhost), so the load balancer will redirect to the same address and the server can answer.

https://httpd.apache.org/docs/2.4/mod/core.html#serveralias