Realtime sessions
A realtime session is one WebSocket that carries audio in (the user's mic, and optionally meeting audio) and converted speech out, plus JSON control messages and events. The SDK wraps all of this; this page documents the underlying protocol.
Lifecycle
- Your server mints a client token (
POST /v1/tokens). - The client connects to
streamUrl?token=…withinexpiresInseconds. - The server sends
session.readywith the audio formats. - The client streams audio and sends control messages; the server streams speech and events.
- The session ends when either side closes, or at
maxMinutes. At 30 seconds before the end you getsession.expiring, thensession.ended, and the socket closes with code4001. - Usage is billed from what was actually used. See usage & billing.
Audio
Client → server: binary frames: [channel: 1 byte][PCM16 little-endian, mono, 16 kHz].
| Channel | Meaning |
|---|---|
1 | The user's microphone. Converted to the selected voice in voice mode. |
2 | Remote/meeting audio. Heard by the agent in agent mode. |
Send 20 ms frames (640 bytes of PCM). Server → client: binary frames of PCM16 LE mono at session.ready.audio.output.sampleRate (24 kHz). Play them in order, without gaps. On a clear event, drop anything buffered.
Control messages (client → server)
type | Fields | Effect |
|---|---|---|
session.update | mode (voice, type, agent), voiceId, language, emotion, speed (0.6–1.5), volume (0.5–2), latency (fast, balanced), agent: { autoReply, bargeIn } | Change settings live |
say | text, emotion?, interrupt? | Speak typed text in any mode |
interrupt | – | Stop all speech immediately |
agent.brief | briefing, name | Set the agent's knowledge and persona |
agent.whisper | text | Private guidance for the agent's next reply |
agent.respond | – | Make the agent speak now |
ping | t | Server replies pong |
voiceId is the voiceId field from GET /v1/voices. The SDK accepts slugs like "leo" and resolves them for you.
Events (server → client)
type | Fields |
|---|---|
session.ready | sid, expiresAt, audio, agentEnabled, catalog, state |
session.updated | state |
transcript | channel (mic, remote), text, final |
speech.start / speech.end | id, source (voice, text, agent), text |
clear | reason (interrupt, barge_in) |
agent.thinking, agent.text, agent.said, agent.suggestion, agent.silent | See agent |
usage | connectedSec, micSec, remoteSec, ttsChars (every 15 s) |
session.expiring / session.ended | secondsLeft / reason |
error | code, message |
Latency
- Typed lines: first audio in about 0.3–0.6 s.
- Live voice: in
fastlatency mode, words are spoken as soon as the live transcript is stable, often before the speaker finishes the sentence. Expect about 0.5–1.2 s end-to-end. balancedwaits for finalized phrases: slightly slower, fewer corrections.
Emotion
emotion accepts: neutral, happy, excited, calm, confident, content, sad, angry, scared, surprised, sarcastic, sympathetic, tired, curious, anxious, disappointed, frustrated, apologetic, flirtatious, mysterious. Each emotion also applies a delivery profile (pace and loudness), so it's audible even on short lines. Emotion is strongest on voices tagged expressive.