We struggled with similar latency issues. We tried these things to reduce our TTFT to 1.1 sec:
1. Self Hosting LiveKit in our region - LiveCloud keep changing your LK region
2. Using Azure's Open AI model - This slashed LLM latency by 50% straight up. Also it's much more consistent now vs Open AI APIs
3. Backchanneling - We backchannel words like "Ok", "Noted" etc.,. this gives a better perceived TTFT.

We actively benchmark our LiveKit agents against Vapi using an open source tool Whispey. We connect both LiveKit and Vapi agents to it and see the comparison to help us better compare the performance.