By Dmytro Yekymov, Senior Software Engineer, Trembit · 29 October 2026
Live translated subtitles in a video call are text of what the other person is saying, recognised and translated while they speak, and shown under the video in the viewer’s own language. Built as a cascade, that is three services in a row: a recogniser, a translator, and a path that gets the text back to the screen. What went wrong here went wrong at the seams of that cascade — in how your code reads what a provider does when a conversation behaves like a conversation. This is a log from one anonymised build, a 1:1 video platform, shipping in production tags since August 2026, and it carries no latency figures.
Key takeaways
Normal conversation closes the recognition stream, twice over — once when the speaker goes quiet, and again at Google’s documented five-minute maximum stream duration. Both looked identical to a dead pipeline until they were classified as reopenable and reconnected silently.
Silence was being counted as failure. In this build the recogniser settled a final on every pause (ours), a conversation is mostly pauses, and each one was counted as a provider failure — so a “Subtitles unavailable” state flashed on and off while the provider was behaving normally. Fixed 13 October 2026.
Translating every interim result is the mistake to avoid. The first design sent every interim guess to the translation API. Interims are now never translated: the overlay echoes the raw source live, and the translation lands only when the utterance settles.
The vendor choice was a data-protection decision — and it was taken before a single subtitle had been rendered.
Product owners: the failure-experience section and “what did we cut” are the two that change a roadmap.
What are live translated subtitles in a 1:1 video call, and what did this build do?
The feature is a participant in the call, not a library in the app. One-to-one audio and video run on
LiveKit Cloud. A subtitle agent joins the same room as a non-visible participant,
subscribes to each speaker’s audio track, streams that audio to Google Cloud Speech-to-Text v2 using
the chirp_2 model, translates the recognised text through Google Cloud Translation v3, and publishes
the result back into the room as data that the clients render as subtitles. Recognition runs in
language-agnostic mode — the service passes languageCodes: ['auto'] rather than naming explicit
locales, Google describes that mode as
“the model automatically infers the spoken language” — but its ['auto'] sample is synchronous,
its streaming sample names a locale, and the page calls streaming language support limited.
Running it on the streaming path is a build fact, not a documented one.
That makes this a cascaded design: transcribe first, then translate, rather than a single speech-to-speech model. Whether that is the right trade-off is argued properly in our write-up on speech-to-speech versus transcribe-then-translate, and this article takes the choice as given.
What this article does not claim: it is one build, not a survey. The client is anonymised and described only as a 1:1 video platform. One thing below is genuinely vendor-specific: the five-minute cap, Google’s documented limit. Translating every interim result was our own design fault, and would be one anywhere. The other two sit in between: the quiet close may be our own sender, and counting silence as failure followed from finals arriving on pauses.
What decided the vendor, and what it does not explain
The decision that shaped this pipeline was a contracting one, and it was taken before a single subtitle had been rendered.
The reasoning we were given is about which vendor. The platform already ran on Google Cloud, so taking recognition and translation from the same provider added no new processor.
That same reasoning was given to us as the answer to a narrower question: why Cloud Translation
rather than Chirp’s own speech translation. Our observation, not theirs: it does not settle that
one. chirp_2 can translate speech itself — documented with the synchronous method, not the streaming
one — and sits with the same provider, so a data-protection argument does not choose between them. No engineering reason for the split was passed to us, and we will not supply one.
Vendor count is contractual, not technical. Under Article 28 of the GDPR, adding a sub-processor is a contractual event, not a configuration change: it needs the controller’s authorisation, and the same obligations imposed by contract. The speech on the call is personal data, and both services process it. Two new vendors means two of those exercises; a provider already in place, with a new purpose, is less.
The timing is the part worth copying. A vendor decision settled before any code depends on it is a different piece of work from the same decision taken after an integration exists.
The pipeline was then built behind a provider-neutral seam: recognition and translation are reached through an interface, so changing provider is a swap rather than a rewrite. If you take one design decision from this article rather than one failure, take that one — speech models and their commercial terms both move faster than the rest of a stack.
What closes a streaming recognition session, and why it looks like a crash
The stream closes on you, for two reasons that are normal in a conversation, and both arrive on the same error channel as a genuine outage.
The stream ends when the speaker goes quiet. What arrived for us was gRPC status 10, ABORTED, with a message about no more client requests —
wording that points at the client side. Ours, not a documented mapping: grpc.io calls ABORTED
a concurrency condition and says nothing about client requests. A recogniser stream is fed by whatever sends audio: if silence stops your sender, the stream ends
because nothing feeds it. Provider-side or your own, silence ends streams, and nothing is
broken.
The stream ends again at the five-minute mark. Google’s limits are explicit: a stream “can remain
open for up to 5 minutes, and the audio must be sent at a rate that approximates real time”
(Speech-to-Text v2 quotas and limits). At that ceiling the stream ended with gRPC status 11, OUT_OF_RANGE — in the gRPC status
codes, “the operation was attempted past the valid
range.” Ours: no Google page we could find names the status for the cap. Any stream running that
long meets it.
One is a documented limit, the other a consequence of how conversations work. Neither is an outage. Both, in a first implementation, look exactly like the recogniser dying: an error object on the stream, no further results, an overlay that stops updating. The fix is not clever, but it has to be deliberate — the two statuses are classified as reopenable, and the agent reconnects a fresh stream rather than surfacing an error.
One scope note, since it is the first thing anyone asks about speech recognition: we hold no evidence on accents, speaking speed or background noise as separate variables, so this article says nothing about them.
Why do translated subtitles rewrite themselves, and what stops it?
Two instabilities stack on top of each other, and only the second one is specific to translation.
The first is the ordinary interim-versus-final problem of any streaming recogniser, and we have written it up separately for same-language captions: as that piece on interim and final results puts it, “Render every interim update in place and captions flicker; show only finals and they lag.”
The second is that translating a sentence that is still growing produces a different kind of churn. This is a published approach with a published cost. Arivazhagan and colleagues at Google describe a re-translation strategy for live captions “where the source is repeatedly translated from scratch as it grows”, at “the cost of incremental instability as the output is continuously refined” (ICASSP 2020). In a translated line that instability is not confined to the tail: a word arriving at the end of a clause can reorder the whole translated sentence, so the viewer sees the line they were halfway through reading rearrange itself.
This build originally translated every interim result. That did two things at once. It flooded Cloud Translation with calls for text that was about to be replaced, and it put a network round trip in front of every single caption update.
The rule now is blunt: interim results are never translated. While someone is speaking, the overlay echoes the raw recognised source text live, as a single line. The translation is requested only when the utterance settles, and it replaces that line once. A viewer who understands the source language gets the live line; a viewer who needs the translation gets whole utterances instead of a sentence that reorders itself under their eyes. The translated line arrives in larger, less frequent steps — a real trade, made because the cost of the alternative was paid on every update.
What does the viewer see when the subtitle pipeline fails mid-call?
The call never drops. That is a structural property of the design rather than a feature: the subtitle agent is a separate participant in the LiveKit room, so its death is not the call’s death. Audio and video carry on exactly as they did.
What the viewer sees is the overlay disappearing and a “Subtitles unavailable” state in its place. The user is told, rather than left to wonder why the text stopped.
What trips that state is the part worth copying, because it has two different triggers on purpose:
Two consecutive utterance-level failures. One failed utterance is not enough to take the overlay away.
A stream-level error that is not one of the reopenable statuses, immediately. No counting, no second chance.
The asymmetry exists because a counting rule alone has a dead spot. If the socket is gone, it will never yield another utterance — so a rule that waits for two failed utterances waits forever, and the viewer sits on a “connecting” state that will never resolve. Counting is the right instinct for transient, per-utterance noise and the wrong instinct for a stream that has stopped existing. You need both rules, and you need to know which class of error you are looking at — which is the same classification work the recognition stream needed above.
What did we cut because it wasn’t good enough?
Two are worth the detail: one feature cut, one bug fixed.
Cut: per-interim translation. The overlay now shows untranslated source text during the live portion of an utterance instead — one call per utterance rather than per revision.
Fixed: silence was being counted as failure. In this build the recogniser settled a final result on every pause — ours, not documented; the RPC reference promises only “zero or one is_final = true result”. A conversation is mostly pauses. Each of those settled finals with no speech in it was being counted as a provider failure — so the two-consecutive-failures rule fired during ordinary gaps in the talking, and “Subtitles unavailable” flashed on and off while the provider was behaving normally. It was fixed on 13 October 2026.
The failure counter has to know the difference between “the provider returned nothing because it broke” and “the provider returned nothing because nobody is talking.” Those are indistinguishable at the level most error handling is written at, and silence is simultaneously the most common state in a conversation and the easiest one to misclassify. If you are building live captions or live subtitles, this is the bug you will write, and it will present itself to your product team as “the subtitles are flickering on and off.”
If you are putting captions and translation inside a video product, this is the stage where the error classification and the failure behaviour get decided — and it is what our WebRTC AI team spends its days on.
What would we decide before building this again?
Four, from the failures above and the vendor decision rather than a checklist.
Enumerate the provider’s error codes before writing any failure handling. Three of the problems here are one problem — code that could not tell a normal provider state from a broken one.
Decide what “failure” means, and make sure silence is not it. Then separate the two trigger classes: count transient per-utterance errors, act immediately on stream-level errors.
Settle the interim rule before designing the UI. Whether interim text reaches the translation stage decides your call volume to a paid API, not just how the caption looks.
Choose the vendor on the data-protection question as well as the model question, then put a seam in.
FAQ
Does Chirp 2 translate speech itself, or do you need Cloud Translation?
Both exist. chirp_2 has its own speech-translation capability — documented with the synchronous
method, not the streaming one — and this build keeps recognition and translation as separate calls. The reasoning we were given for that is a data-protection one. Our own
observation: it does not settle the question, because both sit with the same provider, so we cannot tell you why the stages were kept apart.
Why do live translated subtitles keep changing while someone is speaking? Two causes stack. A streaming recogniser revises its interim guesses as more audio arrives — the same problem as same-language live captions — and on top of that, re-translating a growing sentence can reorder the whole translated line, not just its end. The rule that stops it is to translate only settled utterances. Keeping the text aligned with the video is a third, separate problem, covered in keeping captions in sync with the video.
What happens to the call if the subtitle service goes down? The call continues untouched — the subtitle agent is a separate participant, not part of the media path. The overlay hides and the viewer gets a “Subtitles unavailable” state.
Can live speech translation handle two languages in one sentence?
We do not know, on this build. The documentation says something narrower than the question: Chirp 2’s
language-agnostic mode infers the spoken language and transcribes “in the most prevalent language” —
which we read as being about which language wins rather than about two interleaved languages. Speech-to-Text v2’s RPC reference shows a languageCode on each result and no event that fires when
the language changes mid-utterance. We have no field observation of our own.
Talk to a team that has built this
If you are adding live translated subtitles to a video product, bring the thing you are actually stuck on: which provider errors you should be reconnecting through instead of surfacing, what the viewer should see when the pipeline stops, or a translated line that will not hold still. This is a free 30-minute call with the engineers who build these pipelines — bring your architecture diagram or the failure you are chasing. No deck, no pitch. Tell us what you are building and we will put the right engineer on the call.
