Twilio’s Programmable Video JavaScript SDK
VideoTrack.addProcessor now works on browsers that support OffscreenCanvas as well as HTMLCanvasElement. When used with
@twilio/video-processors v2.0.0, the Virtual Background feature will work on browsers that supports WebGL2. See VideoTrack.addProcessor and @twilio/video-processors v2.0.0 for details.
import { createLocalVideoTrack } from 'twilio-video';
import { Pipeline, VirtualBackgroundProcessor } from '@twilio/video-processors';
const virtualBackgroundProcessor = new VirtualBackgroundProcessor({
pipeline: Pipeline.WebGL2,
// ...otherOptions
});
await virtualBackgroundProcessor.loadModel();
const videoTrack = await createLocalVideoTrack({
width: 640,
height: 480,
frameRate: 24
});
videoTrack.addProcessor(processor, {
inputFrameBufferType: 'video',
outputFrameBufferContextType: 'webgl2',
});
ConnectOptions.preferredAudioCodecs and ConnectOptions.preferredVideoCodecs is not supported by the browser. (VIDEO-12494)Room.getStats() did not reject the returned Promise when an exception was raised while accessing WebRTC stats that due to a TypeError caused by trying to read from the now-removed RTCMediaStreamTrackStats. (VIDEO-12534)Room.getStats() rejects the returned Promise due to a TypeError caused by trying to read from the now-removed RTCMediaStreamTrackStats. Instead, the SDK now reads from the RTCMediaSourceStats. (VIDEO-12411)attach() method of AudioTrack and VideoTrack. (VIDEO-12242)createLocalAudioTrack(). (VIDEO-12383)The LocalAudioTrack and
LocalVideoTrack classes now provide a
new boolean property called isMuted, which lets you know if the audio or video source is currently providing raw media
samples. The classes also emit muted and unmuted events if the value of isMuted toggles. The application can use
these APIs to detect temporary loss of microphone or camera to other applications (ex: an incoming phone call on an iOS device),
and update the user interface accordingly. (VIDEO-11360)
The Room class provides a new method called refreshInactiveMedia,
which restarts any muted local media Tracks, and plays any inadvertently paused HTMLMediaElements that are attached to
local and remote media Tracks. This is useful especially on iOS devices, where sometimes your application's media may
not recover after an incoming phone call. You can use this method in conjunction with the local media Track's isMuted
property described previously to recover local and remote media after an incoming phone call as shown below. (VIDEO-11360)
<button id="refresh-inactive-media" disabled>Refresh Inactive Media</button>
const { connect } = require('twilio-video');
const room = await connect('token', { name: 'my-cool-room' });
const $refreshInactiveMedia = document.getElementById('refresh-inactive-media');
$refreshInactiveMedia.onclick = () => room.refreshInactiveMedia();
const [{ track: localAudioTrack }] = [...room.localParticipant.audioTracks.values()];
const [{ track: localVideoTrack }] = [...room.localParticipant.videoTracks.values()];
const isLocalAudioOrVideoMuted = () => {
return localAudioTrack.isMuted || localVideoTrack.isMuted;
}
const onLocalMediaMutedChanged = () => {
$refreshInactiveMedia.disabled = !isLocalAudioOrVideoMuted();
};
[localAudioTrack, localVideoTrack].forEach(localMediaTrack => {
['muted', 'unmuted'].forEach(event => {
localMediaTrack.on(event, onLocalMediaMutedChanged);
});
});
import { useEffect, useState } from 'react';
export default function useLocalMediaMuted(localMediaTrack) {
const [isMuted, setIsMuted] = useState(localMediaTrack?.isMuted ?? false);
useEffect(() => {
const updateMuted = () => setIsMuted(localMediaTrack?.isMuted ?? false);
updateMuted();
localMediaTrack?.on('muted', updateMuted);
localMediaTrack?.on('unmuted', updateMuted);
return () => {
localMediaTrack?.off('muted', updateMuted);
localMediaTrack?.off('unmuted', updateMuted);
};
}, [localMediaTrack]);
return isMuted;
}
import useLocalMediaMuted from '../hooks/useLocalMediaMuted';
export default function Room({ room }) {
const [{ track: localAudioTrack }] = [...room.localParticipant.audioTracks.values()];
const [{ track: localVideoTrack }] = [...room.localParticipant.videoTracks.values()];
const isLocalAudioMuted = useLocalMediaMuted(localAudioTrack);
const isLocalVideoMuted = useLocalMediaMuted(localVideoTrack);
const isLocalMediaMuted = isLocalAudioMuted || isLocalVideoMuted;
const refreshInactiveMedia = () => {
room.refreshInactiveMedia();
};
return (
<>
...
{isLocalMediaMuted && <Button onClick={refreshInactiveMedia}>
Refresh Inactive Media
</Button>}
...
</>
);
}
This release adds a new feature that preserves audio continuity in situations where end-users change the default audio input device. A LocalAudioTrack is said to be capturing audio from the default audio input device if:
{ audio: true }, or{ audio: { deviceId: 'foo' } }, and "foo" is not available, or{ audio: { deviceId: { ideal: 'foo' } } } and "foo" is not availableIn previous versions of the SDK, if the default device changed (ex: a bluetooth headset is connected to a mac or windows laptop),
the LocalAudioTrack continued to capture audio from the old default device (ex: the laptop microphone). Now, a LocalAudioTrack
will switch automatically from the old default audio input device to the new default audio input device (ex: from the laptop microphone to the headset microphone).
This feature is controlled by a new CreateLocalAudioTrackOptions
property defaultDeviceCaptureMode, which defaults to auto (new behavior) or can be set to manual (old behavior).
The application can decide to capture audio from a specific audio input device by creating a LocalAudioTrack:
{ audio: { deviceId: 'foo' } }, and "foo" is available, or{ audio: { deviceId: { ideal: 'foo' } } } and "foo" is available, or{ audio: { deviceId: { exact: 'foo' } } } and "foo" is availableIn this case, the LocalAudioTrack DOES NOT switch to another audio input device if the current audio input device is no longer available. See below for the behavior of this property based on how the LocalAudioTrack is created. (VIDEO-11701)
const { connect, createLocalAudioTrack, createLocalTracks } = require('twilio-video');
// Auto-switch default audio input devices: option 1
const audioTrack = await createLocalAudioTrack();
// Auto-switch default audio input devices: option 2
const audioTrack1 = await createLocalAudioTrack({ defaultDeviceCaptureMode: 'auto' });
// Auto-switch default audio input devices: option 3
const [audioTrack3] = await createLocalTracks({ audio: true });
// Auto-switch default audio input devices: option 4
const [audioTrack4] = await createLocalTracks({ audio: { defaultDeviceCaptureMode: 'auto' } });
// Auto-switch default audio input devices: option 5
const room1 = await connect({ audio: true });
// Auto-switch default audio input devices: option 6
const room2 = await connect({ audio: { defaultDeviceCaptureMode: 'auto' } });
// Disable auto-switch default audio input devices
const room = await createLocalAudioTrack({ defaultDeviceCaptureMode: 'manual' });
Limitations
MediaClientRemoteDescFailedError was raised when a Chrome Participant who had enabled Adaptive Simulcast (ConnectOptions.preferredVideoCodecs = 'auto') tried to publish a camera Track after publishing a <canvas> Track. (VIDEO-11516)default-src self unsafe-eval are used. (VIDEO-11537)Unhandled Runtime Error: TypeError: null is not an object (evaluating 'el.paused')
logLevel as an optional property. (VIDEO-10659)import keyword was causing problems in webpack and typescript projects. (VIDEO-11220)