I identified and resolved the audio stuttering issue in my decodeAudioData function. The problem was caused by incorrectly calculating the number of samples to be copied into AVAudioPCMBuffer. Initially, I used frameCount as the byte count, but I overlooked that the audio is stereo with two channels. To correctly process the data, the total number of samples must be calculated by multiplying frameCount by the number of channels.

data.withUnsafeBytes { (bufferPointer: UnsafeRawBufferPointer) in
    if let memory = bufferPointer.baseAddress?.assumingMemoryBound(to: Int16.self) {
       inputBuffer.int16ChannelData?.pointee.update(from: memory, count: Int(frameCount * inputFormat.channelCount))
    }
}

Thanks to Gordon Childs for pointing out that I was recreating the audio converter each time. For better efficiency and performance, it's best to instantiate it once in init().