The real latency of a production voice AI agent

A voice agent lives and dies on latency. Get an answer wrong and the caller shrugs; leave two seconds of dead air and they hang up. So when we talk about our real-time phone agent, the number that matters is not a model benchmark. It is how long the caller waits before they hear a human-sounding reply.

The four stages of a turn

Every conversational turn runs through four model stages: speech-to-text, knowledge retrieval, the language model, and text-to-speech. On a GPU box those cost roughly 150 ms for STT, 73 ms for retrieval, 400 ms for the LLM, and 100 ms for TTS. Add them up and you might expect a turn to complete in about 0.7 seconds.

It does not, and this is where a lot of teams get surprised. The measured, end-to-end average is closer to 1.3 to 1.6 seconds on GPU (and 2.4 to 3 seconds on CPU). The gap is everything the benchmarks leave out: telephony media round-trips, voice-activity detection deciding the caller has stopped talking, audio encode and decode, and plain network time between components.

Why it still feels instant

So how do you keep a 1.5-second turn from feeling slow? You do not make the caller wait for the whole turn. We stream the language model's output and split it on sentence boundaries, then speak the first sentence the moment it is ready, while the rest of the answer is still being generated. The caller hears the agent begin talking in roughly 0.4 to 0.8 seconds. Perceived latency, not total latency, is what determines whether a conversation feels natural.

Callers don't measure your full turn. They measure the silence before you start talking.

How to reduce voice AI latency

If you are building voice AI, instrument every stage separately and treat the overhead between stages as a first-class cost. It is often as large as the models themselves. Then optimise for time-to-first-word, not total turn time. That single shift is the difference between an agent that feels robotic and one people forget is a machine.