Managed to figure it out in the end. The samples from PortAudio are interleaved into 1 sample. So one sample contains left and right, I was combining left and right pairs with other left and right pairs.

I needed to split the individual samples from the 'samples' array into two, in order to get the left and right audio streams. Once I got that figured out, I could actually average them into 1 stream by adding them together and dividing by 2 into a single signed int16.