Looks like it was a silly mistake.
The DMA has to be configured using Half Word (16-bit) instead of Word (32-bit) when I transmitting 16-bit audio data (Even though I2S is set transmitting 16-bit data in 32-bit frame). Otherwise the interrupt seems cannot figure out where the midpoint is.