Turned out the issue was with Rider's debugger, that was slowing something down to a crawl. Outside of debugger all the pings were sent in under 100ms and replies received shortly.