A voice AI agent on WebRTC is a real-time pipeline — audio capture, speech processing (either a cascaded STT → LLM → TTS chain or a native speech-to-speech model), and audio playback — running inside a live WebRTC media session. What decides whether it “feels like a conversation” is not the model. It’s the transport layer underneath: how fast the connection establishes, how the jitter buffer handles arriving audio, and whether the agent can stop talking the instant a user interrupts. Most voice-agent guides stop at the model API. This one starts at the SFU, because that’s where the milliseconds and the interruptions you can’t tune away actually live.
What is a voice AI agent on WebRTC, and why does the transport layer matter?
A voice AI agent is a system that carries a spoken conversation with a user in real time: it captures the user’s audio, turns speech into an intent and a response (via STT → LLM → TTS, or a single speech-to-speech model), and plays synthesized audio back. On WebRTC, all of that happens inside a live, bidirectional media session rather than over request/response HTTP.
WebRTC is the transport of choice for a specific reason: it is built for exactly the low-latency, loss-tolerant, bidirectional media a conversation needs. It runs media over UDP with SRTP encryption, manages network jitter with a playout buffer, and degrades gracefully under packet loss instead of stalling the way a TCP stream does. A raw WebSocket audio stream gives you none of that for free.
But the same transport layer that makes WebRTC the right choice also introduces its own variables — variables most voice-agent content ignores entirely. Every downstream problem in this guide has both a model-layer cause and a transport-layer cause. Your agent feels laggy: yes, partly because of STT and LLM time, but also because of jitter-buffer depth and an extra SFU hop. Your agent keeps talking after the user interrupts: yes, partly VAD tuning, but also because buffered outbound audio is still in flight. Existing guides cover the first cause in each pair. The transport-layer cause is what the rest of this piece is about.
Cascading pipeline vs. speech-to-speech — which architecture should you build?
There are two architecture families, and the choice shapes everything downstream.
A cascading pipeline runs three separate stages: speech-to-text (STT) transcribes the user, an LLM produces a response, and text-to-speech (TTS) speaks it. It’s composable — you can swap Deepgram for another STT, or change LLMs, without touching the rest. It’s observable — you can log the transcript, the prompt, and the generated text at each boundary, which makes debugging tractable. The cost is latency and error compounding: every stage adds its own delay, and every stage boundary is a place where a mistake (a mis-transcription, a hallucinated response) propagates forward.
A native speech-to-speech (S2S) model goes audio-in to audio-out in a single model, skipping the intermediate text serialization. It can be lower-latency and more natural-sounding — it preserves prosody, tone, and timing that get flattened when you round-trip through text — but you give up observability and swappable parts. You can’t inspect the transcript between stages because there often isn’t one, and you can’t replace “just the TTS” because there are no discrete stages. OpenAI’s Realtime API is one example of a natively S2S-capable model.
Most production systems in 2026 are still cascaded, primarily because debuggability and per-stage flexibility matter more than raw latency for most products — and because a cascaded pipeline lets you upgrade one component as the field moves without re-architecting. S2S is gaining ground where the last 200 ms genuinely decides the product. We’re describing the direction qualitatively rather than quoting an adoption percentage, because no reliable neutral figure exists for that split.
Whichever family you pick, the transport-layer concerns in the rest of this guide apply identically. A cascaded and an S2S agent both terminate in the same WebRTC media session, hit the same jitter buffer, and face the same barge-in problem.
What is the real latency budget for a voice AI agent — and where does WebRTC fit in it?
The number that matters is glass-to-glass latency: from the moment the user stops speaking to the moment they hear the agent’s first audio. OpenAI’s own architecture work puts the human-feeling range at roughly 500–1,200 ms on the first turn and 300–600 ms on subsequent turns, with a hard ceiling around 2 seconds — past which users assume the call dropped and start talking over the agent.
To hit that, you have to budget every stage. Here’s the breakdown, with the transport term other guides omit made explicit.
| Stage | What it is | Notes on the term |
|---|---|---|
| STT | Time to a usable (partial or final) transcript | Streaming STT emits partials continuously, so the marginal end-of-turn cost is small; batch STT pays the full transcription cost after endpointing |
| LLM first token | Time to the first generated token | You care about time-to-first-token, not full completion — you can start TTS on the first sentence |
| TTS first audio chunk | Time to the first synthesized audio | Streaming TTS starts speaking before the full response is generated; the term is first-chunk latency, not full-render latency |
| Network / transport | The WebRTC term — see below | The part model-layer budgets skip |
The network/transport term is where WebRTC lives, and it has several parts:
- DTLS handshake time. Before any media flows, WebRTC completes a DTLS handshake to establish the encrypted SRTP session. This is a one-time cost at call setup — it affects time to first audio on connection, not steady-state per-turn latency. It matters most for short interactions where setup time is a large fraction of the whole call.
- ICE connectivity establishment. Candidate gathering and connectivity checks add setup time, and whether the path goes direct or through a TURN relay changes the media route length.
- Jitter buffer depth. This is a deliberate trade-off, not a bug. A jitter buffer holds arriving audio briefly so out-of-order and variably-delayed packets can be played out smoothly. A larger buffer smooths a lossy network but adds fixed latency to every packet; too small and jitter causes audible gaps or concealment artifacts. (Live voice audio leans on packet-loss concealment, not retransmission — RTP doesn’t retransmit by default, and NACK/RTX, more common for video, is rarely worth it for speech, where a resent packet arrives too late to use.) Every millisecond of buffer is a millisecond added to both the audio the user hears and the audio your VAD sees.
- Codec and frame size. Opus lets you trade frame size against efficiency: larger frames pack more audio per packet (less overhead) but add encoding/packetization latency; smaller frames cut latency at a bandwidth cost. RFC 6716 defines frame sizes of 2.5, 5, 10, 20, 40, and 60 ms — 20 ms is the WebRTC default, and the 2.5 ms low-delay mode pushes algorithmic delay down toward 5 ms when every millisecond counts.
Then there’s the SFU hop. A peer-to-peer call routes audio directly; an agent behind a Selective Forwarding Unit routes through the SFU, adding a processing/relay hop. Be honest about the magnitude here: for a single call, an SFU hop is real but typically small — order of single-digit to low-tens of milliseconds for one well-provisioned hop — relative to the STT/LLM/TTS terms. WebRTC infrastructure is not usually the dominant latency source for one voice agent — implying otherwise would be wrong. Where the transport layer does become the dominant operational variable is at scale, which is a topology problem covered further down.
Putting numbers on the whole stack: a fast, well-chosen pipeline — for example Deepgram STT, a fast LLM, and Cartesia TTS — can land in roughly the 450–950 ms range. Real-world production medians sit closer to 1.4–1.7 s end to end, with p99 latencies of 3–5 s once you account for cold starts, network variance, and long generations.
The practical takeaway: hitting the “feels human” threshold is a full-stack budgeting exercise, and the transport stage is a line item — not a rounding error you can assume is free.
Does WebRTC add latency compared to a plain WebSocket audio stream?
Yes and no, and the distinction matters. WebRTC’s DTLS/SRTP handshake and ICE negotiation add real one-time setup latency that a raw WebSocket doesn’t have — you pay it at call establishment. But in steady state, once the call is up, WebRTC generally reduces in-call latency and improves resilience: it’s UDP-based, so a lost packet doesn’t head-of-line-block the stream the way it does on a TCP WebSocket, and its jitter buffer is built for real-time media. A WebSocket audio stream over TCP will hold up later audio waiting to retransmit a lost earlier packet — exactly the wrong behavior for a live conversation. That trade — a fixed setup cost for lower and more stable in-call latency — is why voice-agent platforms standardize on WebRTC despite the handshake overhead.
How does barge-in actually work at the WebRTC transport layer?
Why this matters for the product: clumsy interruptions are the fastest way to make a user feel they’re talking to a machine — they talk over the agent, give up, and ask for a human. Crisp barge-in is table stakes for anything that claims to “feel conversational.”
Barge-in is the ability of a user to interrupt the agent mid-sentence and have the agent stop, immediately, the way a person would. Precisely: the system must detect the onset of user speech and halt its own audio output within a very tight window — on the order of ~60 ms — or the interruption reads as broken and unresponsive.
Every competitor guide answers barge-in with “tune your VAD threshold.” That’s the application layer, and it’s necessary but nowhere near sufficient. Here’s what’s actually happening underneath.
Barge-in requires true full-duplex audio at the SFU. The agent has to be listening — processing the inbound RTP stream for voice activity — at the same moment it is speaking, sending outbound TTS over another RTP stream. Both directions are live simultaneously; this is not push-to-talk. That means the agent-side application must react to an inbound audio event (a VAD hit on the user’s stream) and, within tens of milliseconds, truncate and replace the outbound stream it is already mid-way through sending. If your architecture treats the two directions as anything less than fully concurrent, barge-in cannot work.
The jitter buffer taxes barge-in timing directly. A jitter buffer delays playout to smooth network variance — which means two things. First, the audio the user hears trails what was sent by the buffer depth. Second, and more consequentially, the inbound audio your VAD inspects is — in a typical pipeline that runs VAD on buffered, decoded audio — subject to the same buffering before your application ever sees it. (Some architectures deliberately run VAD closer to the wire, on pre-buffer packets, trading jitter noise for earlier detection — a real design lever most teams don’t reach for.) A larger jitter buffer, tuned for a lossy mobile network, literally spends part of your 60 ms budget before detection can even begin. This is the trade-off no application-layer barge-in guide discusses: you cannot simultaneously maximize jitter resilience and minimize barge-in latency — they pull against each other, and you tune the balance against your real network conditions.
Packet loss and Opus DTX confuse VAD. Packet loss can present as silence or as concealment artifacts, producing false negatives (missing a real interruption) or false positives (reacting to reconstructed audio). Worse, Opus DTX — discontinuous transmission doesn’t fully silence the stream: it drops to sparse comfort-noise (SID) packets roughly every 400 ms instead of one frame every 20 ms. So your VAD pipeline has to distinguish three things — DTX-induced sparse audio, genuinely lost packets, and real silence — which is a signal-interpretation problem, not a simple presence/absence check. Treat a DTX gap as “the user went quiet” (or mistake it for packet loss) and interruption handling breaks. We’re describing this qualitatively — there’s no honest single error-rate number to attach — but the mechanism is real and it lives below the VAD threshold you can tune.
Cutting the outbound stream cleanly is more than “stop sending.” When the user barges in, you have to flush audio that is already buffered or in flight — client-side playout buffer and any SFU-side buffering — or the user hears a stale half-second of the agent’s previous sentence after they’ve started talking. A barge-in that halts generation but doesn’t flush already-queued playout still sounds broken. Clean interruption is a stream-truncation problem across the whole media path, not just a “call .stop() on the TTS” problem.
The practical implication: a team that only tunes VAD thresholds and leaves jitter-buffer, codec, and flush behavior untouched will hit a barge-in responsiveness ceiling they cannot tune their way past at the application layer. The last 30–40 ms of a good barge-in live in the transport configuration.
How does turn-taking and endpointing work in a production voice agent?
Why this matters for the product: get this wrong and your agent either cuts customers off mid-sentence or leaves dead air after every reply — both read as “this bot is broken,” and both push people toward a human line.
First, a distinction most guides blur — and blurring it leads to the wrong fix: barge-in is interrupting the agent while it’s talking (the hard part is a fast, clean stop, covered above); turn-taking (endpointing) is deciding the user has finished so the agent can respond, a live problem even when the agent is silent (the hard part is judging completion). A barge-in bug makes the agent talk over you; a turn-taking bug makes it cut you off mid-thought or leave dead air. Conflate them and you’ll reach for the wrong setting.
End-of-turn detection is, at its base, a trailing-silence problem with a genuine trade-off. Set the silence window too short and the agent jumps in during a natural pause — a breath, a word-search, “I’d like to book a flight to… uh…” — and cuts the user off. Set it too long and the conversation drags, with an uncomfortable beat after every user utterance. The usual production tuning window runs from a few hundred milliseconds to a little over a second of trailing silence, but treat any specific number as approximate and implementation-dependent, not a fixed industry standard — the right value depends on your users, your language, and your tolerance for the two failure modes.
Modern systems increasingly go beyond a pure silence timeout. Some combine acoustic silence with semantic and prosodic cues — was that a grammatically complete sentence? Did the intonation rise like a question or fall like a conclusion? — to make an earlier, more confident end-of-turn call than silence alone would allow. This is a direction the field is moving, not a universally solved technique; treat it as an active area rather than a checkbox.
Here’s the transport-layer tie-in that keeps this consistent with the rest of the guide: the audio your endpointing model sees has already passed through the same jitter buffer and codec pipeline as everything else. Turn-taking accuracy is bounded by transport-layer audio quality and delay, not just by the sophistication of the endpointing model. A state-of-the-art endpointer fed jittery, delayed, DTX-gapped audio will still misjudge turns — because the silence it’s measuring has been reshaped by the buffer before it arrives. You can’t out-model a bad media path.
Which leads to the single most useful piece of practical guidance in this section: test turn-taking under realistic network conditions — simulated packet loss, added jitter, mobile-network variability — not just on a clean local connection. Endpointing that’s crisp on localhost and mushy on a real 4G call is the most common way transport-layer problems stay hidden until production.
How does SFU topology affect a voice AI agent at scale
Why this matters for the product: this is the section that decides whether your agent stays fast when you go from 10 users to 10,000 — the difference between a demo that wows and a launch that degrades under load.
At small scale — a handful of concurrent voice-agent calls — topology barely matters. Most reasonable setups work, and the SFU hop is a minor line item in the latency budget. This is why demos never surface the problem.
At production scale — hundreds to on the order of ~1,000 concurrent calls — SFU topology becomes an operational latency and reliability question with three axes:
- Call distribution across nodes. How calls are spread across SFU nodes, and how you rebalance, determines whether any single node becomes a hotspot that adds queuing delay to the calls it’s carrying.
- Where inference runs relative to the SFU. This is the one that quietly re-introduces latency. If the agent’s audio processing (STT, the VAD/endpointing pipeline, the model calls) runs co-located with the SFU node handling that call, the media stays local. If it requires an extra network hop to a separate inference service, you’ve added a round trip inside your own infrastructure — to a budget you spent real effort shaving at the model layer.
- Multi-region routing. How calls route to the geographically nearest node governs the physical distance — and therefore the propagation latency — between a caller and the processing that serves them. A user in Frankfurt served by a node in Virginia pays for that geography on every turn.
The through-line back to the latency-budget section: a poorly planned topology hands back the milliseconds you fought for upstream. Transport-layer discipline has to hold at scale, not just in a demo — the budget you proved on one call is only real if the topology preserves it across a thousand.
One honest limit: we’re deliberately not quoting a capacity number like “an SFU node handles X concurrent voice-agent calls.” That figure depends on codec, video-vs-audio-only, server hardware, co-located inference load, and network — a single number would be misleading. This is architecture-planning guidance, not a benchmark. Where you need a capacity figure, you get it by load-testing your pipeline on your infrastructure, not from a blog.
This is the layer we live in. In one production voice-agent rescue, a team came to us certain their “laggy LLM” was the problem — the real culprit was a jitter buffer tuned deep for a lossy mobile network the agent never actually served, quietly spending a few hundred milliseconds on every turn. That is the kind of bug that hides at the transport layer and never shows up in a model benchmark. Trembit has spent 15+ years there — SDP, ICE, DTLS/SRTP, and SFU architecture across 50+ real-time video and voice systems — and that depth now sits underneath our voice-AI work specifically, not just classic video conferencing. If you’re weighing an SFU topology decision or chasing latency you can’t find at the model layer, our WebRTC AI work is exactly this problem. For a deeper comparison of the SFUs themselves, see our LiveKit vs mediasoup breakdown and our SFU comparison for telemedicine platforms.
Where do frameworks fit — and what do they not solve?
End to end, the path is: WebRTC client ↔ SFU / media server ↔ agent application (the STT → LLM → TTS chain or an S2S model, plus the VAD/endpointing logic) ↔ back through the SFU to the client. Frameworks slot in at the agent-application layer. LiveKit Agents is one production example — it bundles the SFU, real-time transport, and an agent runtime with adaptive interruption handling. Pipecat is another. We name them as reference points, not a feature-by-feature comparison — that’s its own piece.
The point: a framework gives you the agent-application box for free, but it does not absolve you of the transport-layer decisions around it. Jitter-buffer depth, codec/frame-size choice, and SFU topology are yours to get right regardless of which framework fills the middle — and they’re what separate an agent that feels conversational from one that feels like a walkie-talkie. If you’re scoping that layer, Trembit’s AI voice assistants development work starts exactly here.
