STT vs S2S: Choosing a Real-Time Translation Architecture for Your Video Platform

Adding real-time speech-to-speech translation to a live video platform comes down to one architecture decision: cascaded — speech-to-text (STT) → machine translation (MT) → text-to-speech (TTS), three separate models chained together — or end-to-end speech-to-speech (S2S), a single model that takes audio in one language and emits translated audio, with no intermediate text step. Get it wrong and you either over-promise “instant” translation you can’t deliver, or you build the wrong architecture for your scale and compliance needs and pay to redo it. In 2026, cascaded remains the production default for most platforms: broader language coverage, each stage independently debuggable and swappable, and an intermediate transcript you can audit for compliance. S2S trades some of that flexibility for lower round-trip latency and, on some models, a translated voice that preserves the original speaker’s vocal characteristics. This guide gives you a factor-based framework for choosing between the two — and, just as important, the WebRTC/SFU plumbing question that actually determines whether either one works inside a live call.

One boundary up front: this is about translating a live human speaker for live human listeners — a multilingual patient consult, an international lecture, a multilingual town hall. It is not about building a bot that talks back to a user. If your need is a conversational voice agent, the design constraints differ (barge-in, turn-taking, an LLM deciding what to say next), and our guide to voice AI agents on WebRTC covers that job. Here, the human speaker sets the pace; the machine only converts what they’re already saying.

Key takeaways

• Cascaded (STT→MT→TTS) is still the production default in 2026 for platforms that need broad language coverage or independently auditable, swappable stages. End-to-end S2S trades some of that flexibility for lower latency and, on some models, voice preservation.
• The real latency bar isn’t “instant.” The IWSLT simultaneous-translation benchmark caps usable latency at about 2 seconds for speech-to-text and 2.5 seconds for speech-to-speech, and human simultaneous interpreters run a natural 3–5 second delay as their own baseline. The honest target is matching or beating a human interpreter, not achieving zero.
• Google Meet’s production S2S rollout (general availability from 27 January 2026, expanding to mobile through April 2026) shows the architecture works at scale — but it launched with only five Latin-based language pairs, a concrete illustration of the coverage trade-off S2S still carries.
• Picking STT vs S2S is often the easier half of the decision. The harder half is the plumbing: extracting one speaker’s audio off the SFU without disrupting the live call, running inference inside the latency budget, and getting the result back to listeners in sync — as a new audio track or a caption overlay.
• Translated audio is architecturally harder to deliver than translated captions — but not for the reason most people assume. An SFU already does per-subscriber track selection natively; the hard part is that intercepting and re-synthesizing audio inside the SFU turns a stateless forwarder into a media-processing engine, which breaks the assumptions (low CPU per session, no codec dependencies) that make SFUs scale. Translated captions sidestep all of it — they ride the data channel as timestamped text without touching the audio path.
• Trembit has built exactly this, live and in-call. For a healthcare platform, we built real-time multilingual translation inside the video call — a custom Mediasoup SFU with C/C++ audio interception running speech recognition → neural translation → text-to-speech, delivering dubbed audio plus synchronized captions at sub-second latency, HIPAA- and GDPR-compliant with zero data persistence. It draws on two capabilities we’d also shipped on their own — WebRTC interpreter-routing at 4,000+ US hospitals (Martti) and a self-hosted STT→MT→voice-cloning→TTS dubbing pipeline — that our WebRTC AI engineering team fuses per product.

Product and business readers: skip to Cascaded or end-to-end — which should you build? for the decision framework. Engineers who already know the model families: jump to Where does translation run in the media path? — that’s the section most guides skip.

Cascaded vs. End-to-End S2S: What’s the Difference?

A cascaded pipeline chains three separate models. STT transcribes the speaker’s audio into text in the source language; MT converts that text into the target language; TTS synthesizes the translated text back into audio. Each stage is a discrete, swappable component — you can source STT from whichever vendor is best for the input language, MT from whichever handles your language pair best, and TTS from whichever voice engine you prefer. The intermediate text is visible at every hop, which is exactly why the architecture is easy to debug, log, and audit.

An end-to-end S2S model collapses those three stages into one. Audio goes in, translated audio comes out, and there is no exposed text step in between. Fewer hops generally means lower round-trip latency, and because the model works directly on speech, some S2S systems can carry prosody and vocal identity across the language boundary — the listener hears something recognizable as the original speaker rather than a generic narrator. The trade-off: narrower language coverage, less to inspect when something goes wrong, and more dependence on a single model’s maturity.

This is the same architecture split we cover for conversational agents in Voice AI Agents on WebRTC — but the translation lens changes the third stage. In an agent, the final stage is an LLM formulating a novel response; in translation, it’s converting speech that already exists. There’s no barge-in problem in the same sense, because the original speaker’s cadence dictates the pace, not the machine’s. For the SFU-hop, jitter-buffer, and turn-taking mechanics that both jobs share, see that piece rather than re-deriving them here.

How fast does real-time translation actually need to be?

The honest answer is not “instant.” Most vendor marketing collapses two different measurements into a single “feels live” number, and separating them is the whole game.

First-response latency is how long the listener waits before hearing or reading anything at all. Steady-state lag is how far behind the live speaker the translation trails through the rest of the conversation — the gap that persists once output has started flowing. A system can have a snappy first response and still drift several seconds behind by the end of a long sentence, or vice versa. The IWSLT (International Workshop on Spoken Language Translation) 2023 simultaneous-translation shared task ranks systems on translation quality within a fixed latency budget: 2 seconds of Average Lagging for speech-to-text, and 2.5 seconds of Starting Offset for speech-to-speech. (Average Lagging measures the steady-state trail; Starting Offset measures first response — the task caps both, which is the point: usable live translation is a low-single-digit-seconds budget, not zero.)

The reference point that reframes the whole question: a human simultaneous interpreter runs a natural 3–5 second delay — the cognitive time to hear, process, and reproduce. That’s not a failure; it’s the baseline every listener at an interpreted conference already accepts. Published interpreting-technology research also finds interpreters can absorb a couple of seconds of additional system latency before accuracy or fluency degrades noticeably. So the bar for “good enough” isn’t literal zero — it’s matching or beating what a competent human interpreter already does. That’s a more precise, more defensible target than any “sub-800ms” slogan.

Concrete before/after: earlier multi-step audio-translation approaches carried 10–20 seconds of lag, which makes natural back-and-forth impossible — by the time the translation lands, the conversation has moved on. Google’s end-to-end model for Meet reaches near-real-time delivery, which is what makes it usable for live conversation at all. That gap between 10–20 seconds and near-real-time is the difference between a feature people switch off and one they forget is there.

One thing to budget for before inference even starts: WebRTC transport has its own latency floor. The SFU hop and the receiver’s jitter buffer add delay before your translation model sees a single sample — see the latency-waterfall section in Voice AI Agents on WebRTC for that mechanic. And the lower you push latency, the more inference you buy per minute; for the $/minute side of that trade-off, see the cost of running a voice AI platform. This guide is about architecture, not the cost model.

Cascaded pipeline or end-to-end S2S — which should you build?

There’s no universal winner. The decision turns on four factors, and most real platforms weight two or three of them heavily.

1. Language coverage needed. If you’re serving a broad or long-tail set of languages — an accessibility mandate, a genuinely global audience — cascaded wins, because you can source STT, MT, and TTS each from whichever vendor has the best coverage for a given pair. If your language set is narrow and defined (a clinic serving two or three communities, a conference with fixed source and target languages), S2S becomes viable. Google Meet’s launch with five Latin-based pairs is the coverage constraint made visible: a strong S2S model, deliberately scoped.

2. Latency tolerance. For live back-and-forth — a clinical consult, a negotiation — every extra second of lag compounds turn-taking friction, and S2S’s lower round-trip latency earns its keep. For one-directional or lecture-style delivery — an e-learning session, a town hall — the audience tolerates more lag, and cascaded’s flexibility is worth more than shaving a second.

3. Voice preservation and naturalness. Does it matter that the listener hears something recognizable as the original speaker — a patient hearing their own clinician’s warmth, not a flat generic voice? Some S2S models preserve this natively, and a cascaded pipeline can too, if its TTS stage does voice cloning. Here’s a hard-won finding from our own dubbing pipeline: cross-lingual voice cloning is a fundamentally different problem than same-language cloning. Most open-source voice-cloning models learn a speaker representation that entangles vocal identity with the source language’s phonetic patterns — so conditioning the model to speak English from a Ukrainian sample produces a voice that’s recognizably the person but contaminated with source-language artifacts that sound unnatural. The fix was a language-agnostic speaker encoder trained on multilingual data, which separates who the speaker is from what language they’re speaking. That single architectural decision was the difference between output the client would ship and output they wouldn’t.

4. Compliance and data residency. Does the audio need to stay inspectable — a transcript you can review, redact, or retain under a Business Associate Agreement (BAA) — rather than disappearing into an opaque model? Cascaded’s intermediate text gives you exactly that inspection point; pure S2S generally does not expose one. For healthcare and regulated enterprise, that auditability is often the deciding factor on its own. We go deeper on the PHI-flow and BAA-chain mechanics in AI Clinical Scribe in Telemedicine and OpenAI Realtime API + WebRTC in Production.

Where Trembit fits, stated precisely: we have built cascaded real-time translation inside a live WebRTC call, in production. For a healthcare platform, we extended a custom Mediasoup SFU with C/C++ modules that intercept audio at the RTP level, then ran medical-vocabulary speech recognition → healthcare-tuned neural translation → text-to-speech, delivering dubbed audio plus synchronized live captions to listeners at sub-second latency — HIPAA- and GDPR-compliant, with a zero-persistence pipeline that keeps no audio or transcript beyond the processing window. That’s the cascaded architecture this section describes, running live. It built on two things we’d shipped separately first: WebRTC transport and interpreter routing at national healthcare scale (Martti — routing to human interpreters, sub-20-second connection, 4,000+ hospitals) and the STT→MT→voice-cloning→TTS pipeline itself (a dubbing project — real speech-to-speech translation, but batch/offline of pre-recorded video). The honest open questions are about your call pattern — whether per-listener translated audio can hit your latency budget the way captions can, and how your language set and compliance posture shape the build — which is exactly what an architecture review is for.

Where does translation run in the media path of a WebRTC/SFU video platform?

Translation runs in three places along the media path: audio is extracted off the SFU, sent to inference (server-side or a third-party API), and the result is delivered back to listeners as audio or captions. This is the section most vendor comparisons skip, and it’s where the decision actually gets hard. Choosing STT versus S2S is a model question; getting the translated result into and out of a live SFU call is a transport question — and the transport question is usually the one that breaks first.

Extracting the speaker’s audio off the SFU. You can’t run inference on media you can’t reach, and on most SFUs the media is SRTP — encrypted, and not decrypted by default just because you have server access. Pulling one speaker’s audio out for translation needs a deliberate egress path, not casual packet sniffing: a server-side track subscription or a recording-bot / egress pattern that terminates the media, decrypts it, and hands raw audio to your pipeline. This is the same pattern we established (and had reviewer-corrected) in AI Clinical Scribe in Telemedicine — egress / recording-bot / PlainTransport, with SRTP not decrypted by default. If you’re new to the underlying topology, SFU vs MCU vs P2P media-server architecture covers why an SFU (a Selective Forwarding Unit — the standard architecture for scalable group video) forwards rather than re-encodes, which is what makes the next part hard.

Where inference runs. Two options, each with a clear trade-off. Server-side, near the SFU minimizes the network hop — the audio never leaves your infrastructure before translation — which is best for latency and keeps the data flow inside your compliance boundary, at the cost of hosting and scaling the models yourself. A third-party translation API removes the model-hosting burden but adds a network round-trip and, more importantly, a new external party touching the audio — which becomes a compliance question the moment that audio is PHI or otherwise sensitive. Don’t model the cost here; the drivers live in the cost of running a voice AI platform.

Getting the result back to listeners, in sync. This is the split that most surprises teams, because the two delivery modes look similar to a product manager and are wildly different to an engineer:

1. Translated audio, per listener — architecturally hard. The hard part isn’t the routing. A standard SFU already selects which tracks each subscriber receives, so publishing one translated audio track per language and letting listeners subscribe to theirs is ordinary plumbing. The difficulty is upstream of that: to produce those tracks you have to intercept, decode, translate, and re-synthesize audio inside the SFU’s forwarding loop — and the moment you do, the SFU stops being a stateless forwarder and becomes a media-processing engine. The assumptions that make SFUs scale (low CPU per session, no codec dependencies, forward-don’t-touch) no longer hold. On that healthcare build, the fix was to design the translation interception as a lightweight, per-track plugin within the Mediasoup pipeline — enabled only on tracks that need it, so it doesn’t tax the forwarding performance of untranslated tracks in the same session. That has to be engineered in at the C/C++ media layer, not configured on.
2. Translated captions, per listener — comparatively simple. Send the translated text over the WebRTC data channel, timestamp-aligned to the video, and render it client-side in each listener’s chosen language. The audio path is never touched. One inference result can be translated into several target languages as text and delivered as several lightweight data-channel messages, far more cheaply than synthesizing and routing several audio tracks. For a large share of use cases — lectures, town halls, accessibility overlays — captions are not the compromise; they’re the better fit.

Timing and sync. Whichever mode you pick, the output has to stay aligned with the video and not drift from the speaker’s turn — the same jitter-buffer and timing mechanics that govern any real-time media, covered in Voice AI Agents on WebRTC. The practical implication: caption delivery gives you more slack to absorb inference latency without visible drift, while per-listener audio has to hit its timing budget on every segment or the mismatch becomes obvious.

Working through this for a specific call pattern? A 30-minute architecture review will tell you whether captions or per-listener audio fits your product before you commit engineering to either.

Does real-time translation preserve accuracy, accents, and domain terminology?

Machine translation quality has matured faster than public perception for common language pairs — for high-resource pairs, MT is genuinely good and improving. Where it still degrades is predictable: domain-specific terminology (medical, legal, financial), code-switching (a speaker mixing languages or dropping in English technical terms mid-sentence), and less-common accents and dialects. These aren’t edge cases in the settings this piece is about — a clinical consult is dense with domain terms, and a global town hall is full of accents.

Vendors are addressing the terminology problem concretely. DeepL’s voice-to-voice product, launched 16 April 2026, ships a glossary/terminology feature so a platform can pin the correct translation of its own domain vocabulary rather than trusting the model’s default choice. That’s the right shape of fix, and worth asking any vendor about directly.

The honest caveat: accent and dialect handling remains uneven across vendors and language pairs, and no one should promise universal accuracy. Public benchmark numbers rarely transfer cleanly to your specific speakers, languages, and acoustic conditions, so treat any single accuracy percentage with suspicion and test on your audio before you promise a number to stakeholders.

How does real-time translation affect compliance and data residency?

Routing a patient’s or an employee’s audio to a translation service creates a new data flow that has to be mapped into your compliance chain — the same discipline as any other AI vendor hop. If that audio is PHI, the third-party translation vendor is a business associate, and a signed BAA has to cover it before any audio flows; if it’s regulated enterprise data, the equivalent data-processing and residency terms apply. Compliance is a property of the integration, not of the architecture: an S2S model isn’t inherently more or less compliant than a cascaded one, though cascaded’s inspectable transcript does give you a natural redaction and retention point that pure S2S lacks. A caution, though: cascaded gives you the option to inspect the transcript, not a reason to keep it. On that healthcare build, we deliberately retained nothing — every audio buffer and translated string was overwritten continuously, with no swap-to-disk fallback, so no intermediate PHI survived its processing window; the audit log records that a translation happened, not what was translated. In a real-time pipeline there is no “at rest,” so zero-persistence has to be designed in from the start, not bolted on.

Two Trembit precedents are worth distinguishing here, because the compliance shape differs. The healthcare build above is the direct one: real-time AI translation (ASR → NMT → TTS) inside a live HIPAA- and GDPR-compliant call, where the third-party/model boundaries all sit inside the encrypted, zero-persistence pipeline described above. Martti — the WebRTC video-interpretation platform we built across 4,000+ US hospitals under HIPAA and SOC 2 — is a different compliance shape: it routes to human interpreters over WebRTC, not to an AI translation model, so its BAA chain has no model vendor to cover the way an AI-translation integration does. We won’t blur the two — which one your product looks like changes exactly which vendors land in your BAA chain. For the deeper PHI-flow and BAA-chain treatment, see AI Clinical Scribe in Telemedicine; for the specific audio-modality BAA gap on a named S2S vendor, see OpenAI Realtime API + WebRTC in Production.

Choosing the Architecture is Only Half the Decision

Cascaded or end-to-end is the question most guides stop at. The other half — the half that decides whether the feature ships on time — is whether your SFU can actually deliver the translated result back to listeners in sync, as captions or as per-listener audio, inside your latency and compliance budget.

Trembit’s free architecture review looks at your specific call pattern (consult, lecture, town hall), your language and latency requirements, and your compliance posture, and gives you a straight answer on which architecture fits and what the media-path plumbing actually takes to build. It’s a free 30-minute call — bring the specific decision you’re stuck on (STT vs S2S for your language set, captions vs per-listener audio, a latency-budget audit, or the BAA-chain question) and we’ll pressure-test it. No deck, no pitch.

Book a free 30-minute architecture review with the WebRTC AI engineering team that has already built real-time AI translation inside a live, HIPAA-compliant video call.