Pairing with video

Vaux's output is a standard MediaStream audio track, so it combines with any real-time face or character-swap model that gives you a video MediaStream. The result is one persona stream with both a new face and a new voice.

// 1) Video: your face / character model (example: a realtime video SDK)
const camera = await navigator.mediaDevices.getUserMedia({ video: true });
let video;
await videoModel.realtime.connect(camera, { onRemoteStream: (s) => (video = s) });

// 2) Voice: Vaux
const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
let voice;
const session = await vaux.realtime.connect(mic, { voice: 'maya', onAudioStream: (s) => (voice = s) });

// 3) One persona stream
const persona = new MediaStream([...video.getVideoTracks(), ...voice.getAudioTracks()]);
previewEl.srcObject = persona;
persona.getTracks().forEach((t) => peerConnection.addTrack(t, persona));

Lip sync

The converted voice is delayed slightly relative to the user's real mouth movement. For the best result:

  • Feed Vaux's output (not the raw mic) to any lip-sync or avatar model.
  • Use the speaking event to drive talking animations: session.on('speaking', ({ playing }) => …).
  • Keep both sessions alive for the same duration, and end them together.

Billing your users

Pass your user's ID in metadata when minting tokens, then read per-session usage from GET /v1/usage.