I hope this answer can help you.
Verify GPUDirect RDMA Support:
Check if the kernel module nvidia-peermem is installed and loaded.
If it’s missing, you’ll need to install it using NVIDIA’s MOFED software stack.
Test with Host (CPU) Memory First:
Before using GPU memory, test RDMA transfers using regular host memory.
This helps confirm that your RDMA setup and code are working correctly.
Hardware Limitation:
Since your system shows a "NODE" connection, true GPUDirect RDMA is not possible in this configuration.
Unless you can physically move the GPU or NIC to a PCIe slot under the same root complex, you won't get direct GPU-to-GPU transfers.
Current Behavior:
Your code likely performs an RDMA write, but the GPU memory on the receiver side isn’t updated because GPUDirect is not functional.
That’s why the receiver’s GPU buffer shows no change
If you have any further question please let me know.
BR,
Dolle