I found the problem, it's not the hat but some internal configuration in the SIM7600, had the same problem. Fixed it with this command that sets the sample rate for the CODEC.
AT+CPCMBANDWIDTH=1,1
Fixed the problem as you have described.