Per-Speaker Live Captions in Multi-Party WebRTC Calls

Per-speaker live captioning is the job of turning the audio in a multi-party WebRTC call into on-screen text that is attributed to the right participant and rendered in step with the video. Architecturally it is four decisions: where you extract the audio, how you attach a speaker label to it, how you time the overlay, and which accessibility standard you are actually building against. The second one is where most teams go wrong. Vendor speech-to-text docs assume you hand them one audio stream and ask a model to infer who is talking — that is speaker diarization. A call routed through an SFU usually already knows the answer, because each participant publishes a discrete audio track that the server handles separately. A lot of caption implementations mix that away before the audio reaches the STT API, then pay a model to reconstruct the information they just deleted.

Key takeaways

• In an SFU-routed call, speaker identity is often structural — one participant, one published audio track, one identifier the server already holds — not something you need to infer statistically from a mixed stream.
• Deepgram’s own documentation separates multichannel from diarization, and says outright that where only one individual is on each channel, “Diarization would not be useful.” That page never mentions WebRTC, SFUs, or per-participant tracks — the full quotes are below.
• Track separation reduces how much diarization work you need. It does not automatically eliminate it — our own webinar build ran on an SFU and still used diarization in the pipeline.
• Structural attribution breaks at predictable points: shared conference-room devices, SIP/PSTN bridge legs, mute/reconnect track lifecycle, cross-talk inside a single mic, and screen shares carrying system audio. Design the fallback, not just the happy path.
• End-to-end encrypted media rules server-side captioning out entirely — if the SFU can’t read the packets, there is nothing server-side to transcribe. Decide this before you design the pipeline, not after.
• WCAG 2.1 Success Criterion 1.2.4 Captions (Live), Level AA requires that “Captions are provided for all live audio content in synchronized media” — and W3C’s own Understanding document says the criterion was written for broadcast, not to force captions onto ordinary two-way calls. That distinction changes how a webinar product and a two-person consult scope the same requirement.

If you own accessibility or procurement rather than the code, skip to What does WCAG actually require for live video captions? — it is self-contained. Engineers: read top to bottom; the architectural fork is in the second section.

What does “per-speaker captioning” mean in a multi-party WebRTC call?

It means producing a single, correctly-labeled, correctly-timed text layer from N simultaneous audio sources. Three things have to be true at once: the words are right, the name above them is right, and the text appears close enough to the speech that a viewer reading it is still watching the same moment of video.

Most consumer “live captions” features solve a much smaller problem — one person dictating into one microphone. Adding a second speaker changes the problem category, not the difficulty level. You now need an attribution decision for every utterance, and a wrong attribution is worse than a missing caption: it puts words in a named participant’s mouth, which in a clinical consult, a deposition, or a graded seminar is a real liability rather than a cosmetic bug.

Three neighbouring problems get conflated with this one, so it is worth drawing the lines now:

• Captions are not translation. Same-language accessibility captions and translated subtitles share a transcription stage and diverge immediately afterwards. If you are building the translated variant, the cascaded-versus-end-to-end decision is covered in our breakdown of real-time translation architectures for live calls.
• Captions are not scribing. A caption is verbatim, real-time, and for the people on the call right now. An AI clinical scribe is structured, after-the-fact, and for the medical record — a different accuracy bar, since a scribe may summarise while a caption may not silently invent words. If the same audio also feeds documentation, the BAA chain matters, and that is covered in our piece on adding an AI clinical scribe to a telemedicine platform.
• Captions are not agent turn-taking. Captioning is a passive read of the audio. Endpointing and barge-in are control decisions for a voice agent that has to decide when to speak. Different problem, different latency budget.

How is this different from captioning a webinar or a recorded video?

A one-way webinar is the easy end of the same spectrum — one active presenter, controlled audio, an audience that only listens — and recorded video is easier still, since you can take as long as you like and align to a known timeline afterwards. Multi-party is harder on every axis: speakers change without warning and overlap, audio quality varies per participant and device, and there is no post-processing window. The caption is on screen within a second or two of the words or it is not a live caption. Everything below assumes the hard case.

Where do you get the audio — one mixed stream, or one track per speaker?

This is the fork. In a multi-party call routed through an SFU, the server does not mix participants into one audio program — it forwards discrete per-participant streams. Speaker identity is therefore already present in the media plane as a property of the architecture, before any AI touches it.

Now compare that to what a speech-to-text vendor’s docs assume. Deepgram’s own guidance distinguishes two modes plainly: multichannel is “audio that has multiple separate audio channels, and the audio in each channel is distinct,” while diarization is “the process of separating an audio stream into segments according to speaker identity, regardless of channel.” For the case where each person is on their own channel, Deepgram states the conclusion for you: “because only one individual is on each channel, Diarization would not be useful.” (Deepgram, When To Use Multichannel and Diarization)

That page never mentions WebRTC, SFUs, or per-participant tracks — as of our check on 2026-09-07 — which is exactly the gap. The vendor tells you when each mode applies; nobody tells you that an SFU room is already, structurally, the multichannel case.

This isn’t a clever observation — it’s the obvious consequence of reading the SFU’s architecture and the STT vendor’s docs at the same time, which a team starting from the API docs alone never does.

So why do most caption implementations still reach for diarization?

Because STT documentation is written for the general case: a phone recording, a podcast, a downmixed conference-room feed. In all of those the audio genuinely arrives as one stream with unknown speakers, so diarization is the only option available. A builder starting at the API docs inherits that assumption, mixes the SFU’s tracks down to feed the “normal” input shape, then pays — in latency, in per-minute cost, and in attribution errors under cross-talk — for a model to re-derive an identity the SFU handed them for free.

Getting the audio out of the media path is its own exercise, and we walked through RTP interception from a mediasoup SFU in the translation piece linked above. What matters here is what you do with the track identity once you have it: carry it end-to-end as a label rather than discarding it at the mixer.

When does track-identity attribution actually break down?

Often enough that “one track equals one speaker” is a default, not a guarantee. Four cases account for most production failures:

• Shared room devices. A conference-room system publishes one track for everyone in the room, so attribution collapses to “Meeting Room 4” and names need in-room diarization for that leg only.
• SIP/PSTN dial-in legs. A bridged phone leg arrives as one track representing a line, not a person — possibly several people on speakerphone, at narrowband quality that also degrades accuracy.
• Track lifecycle events. Participants mute, reconnect, and switch devices; an implementation that opens one STT connection per track and assumes it lives for the whole call will leak connections or attribute a rejoined participant to a stale label.
• Cross-talk inside one track. Track separation removes cross-participant ambiguity, not acoustic ambiguity within a single mic that picks up the colleague sitting beside them.
• Screen shares carrying system audio. A shared tab or desktop publishes an audio track attributed to the sharing participant, but the voices in it are whoever is in the video being played — attribute it to the participant and your transcript credits them with words they never said.
• End-to-end encrypted media. This one invalidates the whole approach rather than degrading it. If you encrypt media end-to-end — WebRTC insertable streams with SFrame, say — the SFU forwards packets it cannot read, so there is no server-side audio to transcribe at all. Captioning then has to happen on a client that holds the keys, or through a participant explicitly admitted to the session as a transcription endpoint. For anyone building under HIPAA or an equivalent regime where E2EE is on the roadmap, settle this before designing the pipeline: server-side captioning and end-to-end encryption are mutually exclusive by construction, not a tuning problem.

The workable verdict: track identity as the primary signal, per-track voice-activity detection so an open-but-silent track is not labelled as speech, and a diarization fallback for the legs you know are shared or bridged. That is a per-leg decision — the SIP leg can run diarization while the twelve browser participants do not.

How do you keep captions in sync with the video?

Two clocks fight each other. Streaming STT emits interim results fast and revises them as more audio arrives, so early text is unstable; video keeps playing at its own pace and jitter. Render the interim results naively and captions flicker and rewrite themselves under the reader’s eyes. Wait for confirmed text only and captions fall behind the picture.

The pattern that ships is a two-state display: show recent, still-provisional text in a visually distinct style, then promote it to confirmed styling once the model stops revising it. Making the correction visible beats silently rewriting it — more on where we learned that in the next section.

Delivery is the smaller half of the problem: caption text has to reach the client timestamp-aligned with the media, typically over the WebRTC data channel, occasionally a parallel WebSocket, and in some broadcast pipelines burned in server-side. A worked pattern for timestamp-aligned delivery over the data channel is in the translation piece linked above; the general mechanics of carrying AI metadata beside live media — reliability modes, message framing and backpressure — are a topic in their own right.

Building this into an existing product rather than greenfield? That transition — inference inside a media path you already run in production — is what our WebRTC AI practice does day to day; there’s a no-pitch way to pressure-test your plan at the end of this piece.

What does a real per-speaker captioning build actually look like?

Two of ours, with deliberately different architectures.

An accessible webinar platform, built on Jitsi Meet. Live subtitles were a core requirement, not an add-on: the platform generates real-time speech-to-text during sessions and displays “synchronized captions with speaker identification for multi-presenter events,” at a scale of hundreds of concurrent viewers, with post-session searchable transcripts generated from the same live data. (webinar platform case study)

That build ran on an SFU-based architecture and still used speaker diarization as a pipeline stage — kept on as a standing stage for multi-presenter events rather than switched on per problem leg. Our documented lesson from the project was that speech-to-text accuracy is a pipeline problem rather than a model problem: audio pre-processing for varying microphone quality, diarization, punctuation restoration, timing alignment, and graceful handling of corrections, each independently tunable.

So the accurate claim is narrower than “an SFU means you never need diarization.” It is this: track separation reduces how much diarization work you need; it does not automatically eliminate the need for it. Worth being straight about the gap between that build and the per-leg approach recommended above: on a live events platform where any presenter can share a feed or sit in a room with another presenter, running diarization everywhere was the cheaper engineering decision than classifying legs correctly in advance. Selective is the better default; standing-on is the pragmatic choice when you cannot predict which legs are clean.

A real-time transcription system for media production. The contrast case. Here the input genuinely was not pre-separated per-participant tracks — studio, field, and phone-line audio — so diarization was the correct choice, alongside a context engine enforcing entity consistency, retroactive correction, and the confidence-tiered display described above. (real-time speech-to-text case study)

Same capability, two architectures, decided by one question: does the input arrive already separated, or not? That question is the whole design review.

What does WCAG actually require for live video captions — and does it apply to your product?

This section describes standards. It is not legal advice, and nothing here is a determination about your product — take applicability questions to your own counsel or compliance function.

The relevant standard is WCAG 2.1 Success Criterion 1.2.4, Captions (Live), Level AA, which reads in full: “Captions are provided for all live audio content in synchronized media.” (W3C, Understanding SC 1.2.4)

The scope note on the same W3C page is the part product teams rarely read, and it materially changes the conversation: “This success criterion was intended to apply to broadcast of synchronized media and is not intended to require that two-way multimedia calls between two or more individuals through web apps must be captioned regardless of the needs of users.”

Read those two sentences together and the picture is not “all video calls must be captioned.” A one-to-many webinar, lecture, or town hall sits squarely inside what SC 1.2.4 was written for. A two-person consult or tutoring session sits in a zone the standard’s own authors explicitly declined to cover.

To be unambiguous about what that does and does not mean: it is not permission to skip captions. It means this particular criterion is not the thing compelling you. The driver for a two-way consult is user need, your contract, procurement, or a different regulation — and a deaf patient in a telemedicine appointment needs captions regardless of which clause in which standard happens to name the requirement.

WCAG matters commercially because other instruments reference it — the US Department of Justice’s 2024 Title II rule, for instance, states that “WCAG 2.1, Level AA is the technical standard for state and local governments’ web content and mobile apps.” (ADA.gov) If you sell into public sector, healthcare, or education, expect WCAG 2.1 AA in a procurement questionnaire regardless, and expect the follow-up to be about which sessions you caption and how accurately, not whether the feature exists.

Where to start if you’re adding captions to a multi-party call

Start by looking at your own media plane before you shop for a model. If your SFU is already routing one audio track per participant, you have a multichannel problem, not a diarization problem — and the cheapest, most accurate pipeline is the one that never destroys that identity. Then enumerate the legs where the assumption fails (room systems, phone bridges) and design those explicitly instead of discovering them in production. Then decide your display contract: what provisional text looks like, when it gets confirmed, and how far behind the video you will tolerate.

If you want to pressure-test that before you build, book a free 30-minute call with our engineers. Bring the specific decision — your SFU’s track model, the SIP leg you’re not sure how to attribute, the caption lag you can’t get under control, or a WCAG line item in an RFP you need translated into architecture. We’ll work through it with you. No deck, no pitch. If you’d rather start with the capability picture, that’s our WebRTC AI practice.