Your Vapi agent is not broken. It is thinking too slowly.

The report from the business owner is always some version of the same thing: "it works, but sometimes it just goes silent and the caller hangs up."

The instinct is to blame the audio, or swap the voice model. Wrong layer, most of the time.

Here is the actual sequence. The caller speaks. The transcriber handles it fine. The LLM takes too long to produce a response. The endpointing threshold fires while it is still thinking, the call reads as dead air, and it ends. In the Vapi call object you see endedReason: silence-timed-out sitting next to a fat llmLatency value. That field tells you in seconds what a day of guessing would not.

The default config makes this worse than people expect. Vapi's default endpointing sits around 1450ms, which is high next to Retell near 700ms or Pipecat near 300ms. A default assistant config runs roughly 900 to 1200ms p95 turn latency before you have written a single line of business logic. Humans expect a reply in 300 to 500ms and start abandoning past 1.2 seconds. So the default is already sitting at the edge of "feels broken."

And it compounds. Deepgram transcribes in around 150ms. ElevenLabs Flash synthesizes in around 75ms. Those numbers look great in isolation, and then the end to end response takes close to two seconds, because every hop adds tax: STT, endpointing, LLM time to first token, TTS, network. Component benchmarks are marketing. End to end p95 is the number a customer actually feels.

One knob nobody treats as a latency setting: the system prompt. Time to first token scales with prompt size, so every paragraph you bolt on costs milliseconds on every single turn. Keep it byte stable and provider side prefix caching engages. Keep editing it and it never gets the chance to.

The method I actually use:

Start at the call object, not a hunch. It has the status, the ended reason, the per stage latency, the tool call results, and the full transcript. The answer is almost always already in there.

Track p95 per turn, not call averages. Averages hide the one turn where the caller waited three seconds and gave up. The worst turn is the one they remember.

Correlate across providers. Vapi plus Twilio plus your backend. No single log tells the whole story on the hard failures.

The point is not the specific fix. It is that the layer is knowable in seconds if you look at the object instead of reaching for a model swap. Guessing is what burns the days.