My solution was to switch to Windows.Networking.Sockets.StreamSocket. The problem was a single peer to peer connection and it needed to be robust and handle low bandwidths.