This is not an answer but I think it will be helpful to add here as I cannot add a comment. I have a similar issue and had the question for a while here with no answer: MirroredStrategy output varies depending on the visible GPUs

I think as a workaround for now, you can run nvidia-smi topo -m and check the connections between GPUs

In my case, your example works fine when setting CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 but fails if CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7,0