By Stan Reshetnyk, CTO, Trembit · Last updated 21 October 2026
Voice AI latency is the time between the moment a caller stops speaking and the moment they hear the first sound of the agent’s reply. In a cascaded agent it is built from the end-of-turn decision, speech-to-text finalisation, the LLM’s time to first token, text-to-speech time to first audio, network transport and playout buffering, plus the carrier leg for phone callers. These stages overlap rather than simply add up: STT finalises during the silence wait, TTS starts while the LLM is still streaming, and in model-based modes such as Deepgram Flux or OpenAI’s semantic_vad, turn detection happens inside the speech model. Several of the terms are waits, not computation. So when an agent feels about a second slow, don’t start with the model. Check the silence timer, serial hand-offs and the distance between services first.
Key takeaways
- Pin the clock first. Vendors time latency from different start and stop points, and some publish percentiles without saying what was timed.
- The end-of-turn wait is a line item. LiveKit Agents documents a 0.5 s minimum endpointing delay; OpenAI’s Realtime VAD example waits for 500 ms of silence. The wait sets a floor on when a reply can start. Where the LLM starts early, as under LiveKit’s defaults, the two overlap.
- Human turns are fast. In conversation across ten languages, the median gap between a question and its answer is about 100 ms.
- Vendor model latency excludes the network. ElevenLabs’ ~75 ms for Flash v2.5 leaves it out.
- Overlap stages before buying a faster model. Speculative LLM starts and streamed TTS remove waiting, at the price of wasted calls and side effects you must hold back.
- No published number replaces your own, including ours. We haven’t published a measured end-to-end response time for a delivered voice agent. Measure yours per stage, on one clock.
Not the engineer? Read which number to trust and where the missing second goes, then jump to where we’ve dealt with these trade-offs. Engineers: start at the budget template.
Which voice AI latency number should you trust?
Trust a latency number only when you know where its clock starts and stops. The same call produces different figures on each of these five clocks:
- User-to-ear: from the end of the user’s speech to the reply’s first audio at their device. This is what the caller experiences; use it as your headline.
- Turn-declared to first byte: from the system’s end-of-turn decision to the first TTS audio leaving the server. Many agent logs capture this one, because both timestamps live on one machine.
- Component model latency: ElevenLabs’ model docs list Flash v2.5 at “~75ms”, footnoted “Excluding application & network latency”.
- Stage time from a reference point: Artificial Analysis times speech-to-text to the “first Final-denoted transcript shared after SileroVAD detected end of speech”, network included. Reproducible, but the clock starts after end of speech is already detected.
- Undefined: Twilio’s ConversationRelay page cites “p50 491 ms, p95 713 ms” from internal benchmarks, adding “Results may vary”, with no start or stop point.
The gap between clocks 1 and 2 is where the endpointing wait, both network legs and the playout buffer hide. A team reporting clock 2 can honestly show a fast figure while callers hear a slow one.
As a human reference, Stivers et al. measured a median of +100 ms (mean +208 ms) between a question ending and its answer starting, across ten languages (PNAS, 2009). That describes people talking to people, not a requirement for AI, but it explains why half a second of silence registers.
In plain terms: ask for p50 and p95 user-to-ear latency, the region, and whether phone calls are included. If nobody can say where the clock starts, that is your first finding. It’s the same question we suggest putting to any voice AI vendor or integrator.
What does a voice AI latency budget look like, stage by stage?
A voice AI latency budget lists every stage between the user’s last word and the agent’s first sound, each with a measured number. This template has no millisecond ranges on purpose: per-stage figures depend on your models, regions and settings, and published ranges rarely share a method.
| Stage | What it waits on | Where to read your number |
|---|---|---|
| Capture and encode | Frame size, codec lookahead (Opus algorithmic delay runs “from 5 ms to 65.2 ms”, RFC 6716) | Client timestamps |
| Uplink and jitter buffer | Network path, agent-side buffer depth | WebRTC stats on the agent or server side |
| End of turn | Silence timer or turn model | Last voice frame vs. turn declared |
| STT finalisation | Final transcript (part of end of turn where the STT detects turns) | Turn declared vs. final transcript |
| LLM time to first token | Prompt size, tool calls, region | Request sent vs. first token |
| TTS time to first audio | First synthesised chunk | First token vs. first audio byte |
| Downlink and playout | Network path, client jitter and playout buffer | Browser getStats: jitterBufferDelay ÷ jitterBufferEmittedCount |
| Telephony leg | Carrier, SIP gateway, narrowband audio | Same agent: phone call vs. web call |
Jitter-buffer stats sit on the receiving side’s inbound-rtp report. A browser’s getStats therefore shows the downlink buffer, agent audio arriving at the client. The uplink buffer lives in the agent’s or server’s WebRTC stack, and may not exist if an SFU forwards without buffering. The W3C WebRTC statistics spec defines jitterBufferDelay as the summed time samples spend in the buffer, so divide by jitterBufferEmittedCount for a per-sample average. Rows overlap, so read the table as a timeline, not a sum.
The one row you can fill from documentation is end of turn. On LiveKit Agents’ defaults, the agent waits at least 0.5 s after speech stops before declaring the turn over. LiveKit also enables preemptive generation by default, so the LLM can start during that wait. The cost is then roughly the longer of the two, not their sum: with a slow model the timer is hidden; with a fast one, the timer is what the caller hears.
Transport mechanics (ICE, DTLS, the SFU hop, flushing audio on barge-in) are covered in how the transport layer, barge-in and turn-taking work for a WebRTC voice agent.
Where does the missing second actually go?
Four places, starting with the one no model benchmark shows.
Which latency source should you check first?
The end-of-turn wait, because it is invisible to every model benchmark. It isn’t always the largest term; a tool-call round trip or a cold LLM can cost more. The documented settings, read from each vendor’s docs on 21 October 2026:
| Stack | Documented end-of-turn setting |
|---|---|
| LiveKit Agents | Endpointing min_delay 0.5 s, max_delay 3.0 s |
| Pipecat | VAD stop_secs 0.2 s plus Smart Turn, as the default configuration |
| OpenAI Realtime | server_vad example: silence_duration_ms 500 |
| Deepgram Flux | Model-based: eot_threshold 0.7, eot_timeout_ms 5000 |
Read each in scope. LiveKit’s 0.5 s is a minimum in its default fixed mode; the Python SDK also offers a dynamic mode that adapts to pause patterns. OpenAI’s 500 ms is an example, not a stated default. Pipecat pairs its 0.2 s VAD stop with Smart Turn, which makes the main end-of-turn call. Deepgram’s 5000 ms is a timeout ceiling, not a typical wait.
In plain terms: every agent waits for a set amount of silence, or a model’s verdict, before it answers. That wait is a setting someone chose, so ask what yours is.
Log the last voice frame and the turn-declared timestamp; the gap is your endpointing cost. If you’re still choosing a framework, see our Pipecat vs LiveKit Agents comparison.
Sequential hand-offs
When STT, LLM and TTS run one after another as separate services, each boundary is a network request. OpenAI’s latency optimization guide puts the LLM-side version plainly: “Each time you make a request, you incur some round-trip latency—this can start to add up.” Tool calls compound it: a CRM lookup means a tool request, the tool’s own latency and a second model call. Count the serial requests per turn in a trace.
Region hops and cold starts
Every service in a different region from the agent host adds distance to every request: an agent in Frankfurt calling STT, LLM and TTS in three other regions pays it several times per reply. List where each component runs and measure the delay from the agent host to each; placing inference next to the media path is covered in the SFU topology section of our WebRTC voice agent guide. The first turn also pays for work later turns skip: provider connections, TLS handshakes and, on self-hosted models, warm-up. Split first-turn from steady-state metrics; if the first is consistently slower, pre-warm.
Buffering at both ends, and the telephony leg
Buffers trade delay for smoothness: the receiving jitter buffer grows on unstable networks, and the client playout buffer adds its own margin. Phone calls also cross a carrier network and a SIP gateway as narrowband audio; compare user-to-ear latency for the same script on web and phone. For the PBX side, see our FreeSWITCH voice agent guide; for media-path work, see our WebRTC and AI engineering.
How do you measure end-to-end latency so the number is believable?
- Headline: user-to-ear. Record a test call at the client with both sides on one track and read the gap off the waveform. Log turn-declared-to-first-byte on the server beside it; the difference is your endpointing, network and playout share.
- Five timestamps per turn: last user voice frame, turn declared, first LLM token, first TTS byte, first audio at the client. Where turn detection runs inside the STT or speech model, “turn declared” and the final transcript arrive together.
- p50 and p95, not the mean, split into first and steady turns and phone and web callers, on the networks and regions you ship to.
- Re-measure after any endpointing change. Pipecat notes that its built-in STT P99 values “are measured with
stop_secs=0.2“. Change that setting and the published figure no longer describes your system.
How do you claw the second back?
Start with levers that remove waiting; they’re usually cheaper than a faster model. Ordered roughly from least to most disruptive:
| Lever | What it removes | What it costs or risks |
|---|---|---|
Smarter end-of-turn decision: a learned turn model (Pipecat’s Smart Turn, LiveKit’s turn detector, OpenAI’s semantic_vad), not a longer silence timer |
Silence-timer wait | Cut-offs if too aggressive |
| Start the LLM before the turn is confirmed | LLM work after the turn ends | Wasted calls, premature side effects |
| Stream LLM output into TTS by phrase | Waiting for full completion | Phrase-boundary handling |
| Stable prompt prefix for caching | Input processing before the first token | Prompt-structure discipline |
| Co-locate services in one region | Cross-region round trips | Provider and region limits |
| Pre-warm provider connections | First-turn setup | Idle connections |
| Match transport to the caller | Unneeded hops | Two media paths to run |
| Speech-to-speech model | STT and TTS hand-offs | Visibility, flexibility, cost |
Speculative LLM start. LiveKit’s docs note that its default-on preemptive generation “doesn’t always reduce latency” and leave preemptive TTS off because of “wasted compute on cancellations.” Deepgram’s Flux quickstart warns that starting the LLM on EagerEndOfTurn “can increase LLM API calls by 50-70%”.
The bigger risk is correctness. Of a server-side function that already reached your endpoint, Deepgram’s Voice Agent docs on speculative replies say: “Deepgram discards the response, but your endpoint ran.” A speculative turn can book an appointment or write to a CRM on a transcript the user then changes. Let speculative turns call read-only tools only, and hold side-effecting calls until the turn is confirmed, keyed by turn ID so a resumed turn can’t fire twice. Deepgram’s page recommends the same deferral.
Prompt caching. OpenAI’s prompt caching guide says caching is on by default but “requires the entire rendered prefix to match”, so keep the system prompt and tool definitions first and unchanged.
What not to do first: swap in a faster LLM before measuring the end-of-turn wait. A faster model shortens one term. If the silence timer is the bigger term, you’ve paid for the wrong fix.
Stuck at a number you can’t explain? A free 30-minute call with a Trembit real-time engineer is the place to walk through your per-turn timestamps.
Does a speech-to-speech model remove the latency problem?
It removes some terms, not the problem. A speech-to-speech model such as the OpenAI Realtime API drops the STT and TTS hand-offs and the text round trip. The end-of-turn decision, transport, telephony leg and region distance remain. OpenAI’s semantic_vad still makes you choose between low, which “will let the user take their time to speak”, and high, which “will chunk the audio as soon as possible.” You also lose per-stage visibility and the freedom to swap components. We cover what S2S versus cascaded changes on the invoice and running the Realtime API over WebRTC in production separately.
Where have we dealt with these trade-offs?
We haven’t published a measured end-to-end response time for a delivered voice agent, and we’re not going to invent one here. What we can show is the same trade-offs, buffering against delay and silence against interruption, in three systems we built. Trembit has delivered 50+ video and voice projects, and our founders have worked together on video and voice since 2009. In the second system below, the latency work sat in the media path, not the model.
An AI phone-screening agent for a staffing company. The bot places outbound screening calls, so it lives with the telephony leg and the end-of-turn problem at once. Our case study records what went wrong first: “Early versions of the bot had technically accurate transcription but awkward timing — it would start speaking before the candidate finished a thought, or wait too long after they stopped, creating uncomfortable silences.” The fix was a multi-signal turn-taking model combining audio energy, speech cadence, the syntactic completeness of the transcript and prosodic cues such as falling intonation. Call completion improved, and candidates stopped hanging up mid-screening. That is a completion effect; the case study gives no latency figure.
In-call medical translation inside a telehealth video platform. We worked inside the media path here: native modules in a custom mediasoup SFU intercept audio at the RTP level, and adaptive buffering balances translation latency against naturalness, “accumulating just enough speech for accurate phrase-level translation while keeping the perceived delay under one second for typical clinical dialogue.” That is a perceived-delay design goal for translation, not a measured agent response time. More buffer buys accuracy and costs naturalness.
Real-time speech-to-text for a media company. Live captioning needed partial results “within 1–2 seconds of speech”, on on-premises GPU inference. The answer was streaming decoding: emit partial transcripts as speech progresses, show high-confidence words immediately, and hold uncertain segments until more audio resolves them. Note the scope: 1–2 s is a captioning figure, not an agent’s STT-finalisation time; the streaming technique is what carries over.
Frequently asked questions
What is a good end-to-end latency for a voice AI agent? There is no published standard for AI agents. The closest neutral reference is human conversation, where the median gap between a question and its answer is about 100 ms. Set your target from your use case, then measure user-to-ear p50 and p95 against it.
Why is my voice agent slow even with a fast LLM? Because the LLM is one term of several. The usual causes are the end-of-turn wait, serial hand-offs between STT, LLM and TTS, services in different regions, and timing full completion instead of time to first token. The five per-turn timestamps above show which.
What is the difference between time to first audio and end-to-end latency? Time to first audio usually describes the TTS stage, sometimes excluding network. End-to-end latency, measured user-to-ear, runs from the user’s last word to the first sound they hear, including endpointing, transport and playout. See which number to trust.
Is a speech-to-speech model always faster? It removes the STT and TTS hand-offs but keeps the end-of-turn decision, transport and region distance. Whether it’s faster for you depends on which terms dominated before.
If your agent feels slow and the logs don’t say why, book a free 30-minute call with a Trembit real-time engineer. Bring your per-turn timestamps (last user voice frame, turn declared, first LLM token, first TTS byte, first audio at the client), your endpointing settings, and the region of each hop. By the end of the 30 minutes you’ll have a ranked shortlist of where your missing second most likely goes and the first lever to try. No deck, no pitch. Start from our voice AI agent development page.
