WebRTC: Leveraging New Features October 15, 2019 Website: - - PowerPoint PPT Presentation

WebRTC: Leveraging New Features

October 15, 2019

WebRTC, Mobility, Cloud, IOT and More...

IIT REAL-TIME COMMUNICATIONS

Conference & Expo Oct 14-16, 2019 Chicago

Bernard Aboba

Principal Architect at Microsoft

https://webrtc.internaut.com/iit-2019/ Website:

Goals for Today

• To introduce you to shipping features of WebRTC standards:
• WebRTC 1.0 Candidate Recommendation
• Screen Capture
• Content-Hints

WebRTC Standards

Shipping

WebRTC 1.0: Recycled at Candidate Recommendation (in all browsers) WebRTC-Stats: Recycling (soon) at Candidate Recommendation (in all browsers) Media Capture and Streams: Recycled at Candidate Recommendation (in all browsers) Screen Capture: Working Draft (in all browsers) Audio Output Devices: Candidate Recommendation (in Chrome and Edge Beta) Media Capture from DOM Elements: Working Draft (in Chrome and Edge Beta) MediaStream Recording: Working Draft (in Chrome and Edge Beta) Content Hints: Editor’s draft (in Chrome and Edge Beta)

Completed Origin Trial (Chrome and Edge Beta)

P2P QUIC: Editor’s draft in ORTC CG P2P QUIC Origin Trial Announcement

Not yet implemented

DSCP Control API: Editor’s draft WebTransport: Editor’s draft in WICG WebRTC-SVC: Editor’s draft WebRTC-NV Use Cases: Working Draft

The Evolution of WebRTC

Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API)

• (widely varying) SDP only, no realtime object model
• no longer included in the WebRTC specification

2. addTrack and removeTrack (implemented)

• Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox)
• Support for RtpSender/RtpReceiver objects

3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API)

• Required: “Unified Plan” SDP
• Additional object model support
• Advanced features: simulcast, direction, etc.

Blog by Jan-Ivar Bruaroey of Mozilla:

https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

Why is there an object model in WebRTC 1.0?

• Need a way to tweak parameters on individual tracks sent over the wire
• Bitrate
• Framerate
• Direction (sendonly/recvonly etc.)
• Existing control surfaces insufficient
• createOffer params - not per-track
• AddStream params - not modifiable post-add
• MediaStreamTrack constraints - affects raw media, not encoding

PeerConnection PeerConnection JavaScript Application (Sender) JavaScript Application (Receiver)

Track

Track

Internet

Track

Track SDP SDP

Tracks handle raw media No per-track API surface

The Evolution of WebRTC (cont’d)

Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API)

• (widely varying) SDP only, no realtime object model
• no longer included in the WebRTC specification

2. addTrack and removeTrack (implemented)

• Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox)
• Support for RtpSender/RtpReceiver objects

3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API)

• Required: “Unified Plan” SDP
• Additional object model support
• Advanced features: simulcast, direction, etc.

Blog by Jan-Ivar Bruaroey of Mozilla:

https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

The Evolution of WebRTC (cont’d)

Since its conception in 2011, the WebRTC API has gone through three major iterations: 1. addStream and removeStream (widely implemented legacy API)

• (widely varying) SDP only, no realtime object model
• no longer included in the WebRTC specification

2. addTrack and removeTrack (implemented)

• Both “Plan B” (Chrome, Edge) and “Unified Plan” SDP (Firefox)
• Support for RtpSender/RtpReceiver objects

3. addTransceiver (Favored WebRTC 1.0 Candidate Recommendation API)

• Required: “Unified Plan” SDP
• Additional object model support
• Advanced features: simulcast, direction, etc.

Blog by Jan-Ivar Bruaroey of Mozilla:

https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

addTransceiver Concepts

• Implemented first in Firefox 59, now available in Chrome (simulcast support in M73+)
• addTransceiver() creates a RTCRtpTransceiver object with associated

RTCRtpSender/RTCRtpReceiver objects.

– Allows RTCRtpSender to be initialized with sendEncodings, establishing the “simulcast envelope” (number of streams, order of encodings). – setParameters() cannot change “simulcast envelope” though it can set individual streams to active/inactive.

• RTCRtpTransceiver has an m-line identifier (mid) attribute that identifies the

transceiver uniquely on both the local and remote peer.

– Track.id rarely correlates due to RTCRtpSender/RTCRtpReceiver model. – Implies that each m-line corresponds to one and only one RTCRtpReceiver/Sender pair. Only true in “Unified Plan” SDP.

Blog by Jan-Ivar Bruaroey of Mozilla: –https://blog.mozilla.org/webrtc/rtcrtptransceiver-explored/

WebRTC 1.0 CR: Now in Chrome, Edge Beta, Firefox and Safari

• WebRTC 1.0 CR + “Unified Plan” is now the default in all

browsers.

– Chrome Announcements: – https://groups.google.com/forum/#!topic/discuss-webrtc/P9qHa0inKtY – https://groups.google.com/forum/#!topic/discuss-webrtc/aPRI0-l7lvY – Chrome Rollout began in late January 2019:

https://chromiumdash.appspot.com/schedule

• Transition guide:

– https://docs.google.com/document/d/1-ZfikoUtoJa9k-GZG1daN0BU3IjIanQ_JSsc

HxQesvU/

What You Need to Know

• To retain Plan B behavior in Chrome:

– new RTCPeerConnection({sdpSemantics:'plan-b’});

• To explicitly ask for Unified Plan (now the default):

– new RTCPeerConnection({sdpSemantics:'unified-plan’});

• Feature detection

– To check for addTransceiver support:

if(typeof(RTCRtpTransceiver) !== "undefined"){

• utput('RTCRtpTransceiver supported\n’);

}

– Chrome specific: pc.getConfiguration() returns the value of sdpSemantics.

• Read the Transition Guide!!

WebRTC 1.0 Object Model

PeerConnection

RTPSender

PeerConnection JavaScript Application (Sender) JavaScript Application (Receiver)

Track

Track

RTPSender

DTLS Transport ICE Transport Internet

ICE Transport

DTLS Transport

RTP Receiver

RTP Receiver

Track

Track SDP SDP

RTPSender

DataChannel

RTPSender

SctpTransport

RTPSender

DataChannel

RTPSender

SctpTransport

WebRTC 1.0 Object Model

• RtpSender* (Section 5.2)
• RtpReceiver (Section 5.3)
• RtpTransceiver (Section 5.4)
• DtlsTransport (Section 5.5)
• IceTransport (Section 5.6)
• SctpTransport (Section 6.1.1)
• DataChannel (Section 6.2)
• DTMFSender (Section 7.2)

* all objects are prefixed with “RTC”

Specification: https://w3c.github.io/webrtc-pc/

WebRTC 1.0 specification additions

PeerConnection .getSenders() .getReceivers() .getTransceivers() .addTrack() .removeTrack() .addTransceiver() .sctp ... RtpSender .track .transport .getCapabilities() .getParameters() .setParameters(params) .replaceTrack(track) .setStreams() ... RtpReceiver .track .transport .getCapabilities() .getParameters() .getContributingSources() .getSynchronizationSources() .getStats() ... DtlsTransport .iceTransport .state .getRemoteCertificates() .onstatechange .onerror ... IceTransport .role .component .state .gatheringState .getLocalParameters(), .getRemoteParameters(), .getLocalCandidates() .getRemoteCandidates() .getSelectedCandidatePair() .onstatechange .ongatherinstatechange .onselectedcandidatepairchange ... SctpTransport .transport .state .maxMesageSize .maxChannels .onstatechange

RtpParameters

.headerExtensions .rtcp .codecs RtpSendParameters: RtpParameters .encodings .degradationPreference .priority RtpReceiveParameters: RtpParameters .encodings RtpCodingParameters .rid RtpEncodingParameters .rid .codecPayloadType .dtx .active .ptime .maxBitrate .maxFramerate .scaleResolutionDownBy IceParameters (read only) .usernameFragment .password

Source: https://w3c.github.io/webrtc-pc/

What can you do with WebRTC 1.0 objects?

• "Warm up" media path while the getting a track and ringing
• Change the send codec (without SDP munging)
• Change the camera source instantly (front to back)
• Enable/disable sending of media instantly (without signalling)
• Set a maximum bitrate or maximum framerate
• Obtain detailed status of individual ICE and DTLS transports
• Send simulcast
• Receive simulcast (optional)

WebRTC API Simulcast Examples

Source: https://w3c.github.io/webrtc-pc/#rtcrtpencodingspatialsim-example* Note: maxFramerate is not yet implemented in any browser.

Demo: Simulcast Playground by Philipp Hancke

https://fippo.github.io/simulcast-playground/chrome

Note: Currently, simulcast is not yet interoperable between browsers. As a result, this demo only runs on Firefox and Chrome or Edge Preview “Plan B”. Chrome/Edge “Unified Plan” does not surface SSRCs when simulcast is enabled, unlike Firefox. A fix for Chrome to resolve this issue is specified here: https://tools.ietf.org/html/draft-alvestrand-mmusic-simulcast-ssrc

Implementation Status

• Web-platform-tests dashboard “does not contain useful metrics for

evaluation or comparison of web platform features”

• Web confluence project:

○ Looks at properties and methods exposed by browsers: ○ https://web-confluence.appspot.com/#!/ ○ Caveat: no guarantee that a widely-supported API is interoperable in its details, or will remain part of the web platform. ○ Tool that extracts data from the confluence tracker: https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc

• Overall status:

○ Fewer features with no implementations, compared with TPAC 2018. ○ All objects now supported in Chrome and Edge Preview

17

WebRTC 1.0 Implementation Status

https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc

Features “with intent to implement”:

RTCRtpEncodingParameters.maxFramerate

Features “at risk”:

RTCPeerConnection.getDefaultIceServers(), RTCCertificate.getSupportedAlgorithms(), RTCRtpEncodingParameters.dtx RTCRtpEncodingParameters.ptime

WebRTC 1.0 Implementation Status (cont’d)

WebRTC 1.0 Implementation Status (cont’d)

WebRTC 1.0 Implementation Status (cont’d)

Coming Attractions: Recent Intent Statements

(from https://groups.google.com/a/chromium.org/forum/#!forum/blink-dev )

• Intent to Implement: WebRTC Insertable Streams (10/1/19)
• Intent to Implement: QuicTransport (10/4/19)
• Intent to Ship: RTCRtpREceiver playoutDelayHint property (10/4/19)
• Intent to Experiment: WebSocketStream (9/5/19)
• Intent to Implement: Screen Enumeration (9/12/19)
• PSA: Private IP Addresses exposed by WebRTC changing to mDNS hostnames (8/13/19)
• Intent to Implement: WebRTC SVC extensions (7/21/19)
• Intent to Implement & Ship: WebRTC Perfect Negotiation APIs (7/22/19)
• Intent to Implement & Ship: RTCPeerConnection::onicecandidateerror (6/14/19)
• Intent to Implement and Ship: RTCRtpTransceiver.stop() (4/8/19)
• Intent to Implement and Ship: Simulcast with PeerConnection AddTransceiver API in WebRTC (3/27/19)
• Intent to Implement and Ship: SctpTransport in WebRTC (2/25/19)
• Intent to Implement: RTCError, RTCErrorEvent, RTCErrorEventInit (1/21/19)
• Intent to Implement and Ship: RTCRtpEncodingParameters.scaleResolutionDownBy &

RTCRtpEncodingParameters.maxFramerate (1/18/19)

• Intent to Ship: WebRTC RTCDtlsTransport object (1/11/19)
• Intent to Implement and Ship: RTCRtpReceiver.getParameters() (11/27/18)
• Intent to Implement and Ship: RTCPeerConnection.connectionState and
• RTCPeerConnection.onconnectionstatechanged (11/22/18)
• Intent to Implement and Ship: RTCRtpTransceiver.setCodecPreferences (10/2/18)

The Evolution of Screen Sharing

Web-based screen sharing has evolved from a model based on whitelists and extensions, to standardized getDisplayMedia and content-hints APIs that “just work”. Supported in Edge, Chrome, Firefox, and Safari (preview). Specification: https://w3c.github.io/mediacapture-screen-share/ Blog: https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen-capture-to-micros

• ft-edge-media-capture-api/

Screen Capture API

Supported browsers

Example Usage

var constraints = { video: true }; function handleSuccess(stream) { video.srcObject = stream; } function handleError(error) { console.log('navigator.MediaDevices.getDisplayMedia error: ', error); } navigator.MediaDevices.getDisplayMedia(constraints). then(handleSuccess).catch(handleError);

Demo

https://webrtc.internaut.com/iit-2019/screen-capture/

Content Hints API

Goal: Enables optimized content handling. For example, when capturing the screen of an application, the “detail” hint can be provided (“motion” would be more appropriate for a video of a soccer game).

Content Hints Audio Values

Content Hints Video Values

Demo

https://webrtc.github.io/samples/src/content/capture/video-contenthint/

Thank you!