Just to mention that since nginx 1.27.3 in a non commercial version of nginx resolver and resolve argument for server may be used in upstream block. This allows to limit addresses to ipv4 only using resolver IP ipv6=off in the upstream block.

More details in the related serverfault answer and in nginx upstream module docs