Okay so turns out one solution to this is to just let Pion take care of the RTP packet sending. Instead of sending my video to a port over UDP using gstreamer's udpsink, I changed the application to use appsink (also gstreamer). I am now writing the data samples to the peer connection tracks, instead of manually creating RTP packets and writing them.