Okay. After 2 weeks of searching I have found a solution to this issue.
Seems in RNCallKeep library when you end the call if you used normal RNCallKeep.endCall()
function, there may be some AudioSession can be hanging there and cause audio issues for the next calls. So my fix was calling RNCallKeep.endAllCalls()
instead. So then this will clear all the audio sessions and restore the default configurations. This was fixed my audio issues.