WebRTC & ORTC Tutorial October 14, 2019 Tutorial website: - - PowerPoint PPT Presentation

WebRTC & ORTC Tutorial

October 14, 2019

WebRTC, Mobility, Cloud, IOT and More...

IIT REAL-TIME COMMUNICATIONS

Conference & Expo Oct 14-16, 2019 Chicago

Bernard Aboba

Principal Architect at Microsoft

Robin Raymond

CTO at Optical Tone

https://webrtc.internaut.com/iit-2019/ Tutorial website:

Goals of Today’s Tutorial

• To introduce you to the basic concepts of Web audio/video input and
• utput:
• Tracks and Streams
• Capture and Output APIs
• To introduce the realtime object model used in the WebRTC and ORTC

APIs

• To demonstrate how the object model can be used to build simple as well

as advanced applications.

Data Channel

Rtp Sender Track

ORTC Object Model

Application Application Track Rtp Sender

Dtls Transport Ice Transport Ice Gatherer Ice Transport Controller Ice Gatherer Ice Transport Dtls Transport

Rtp Receiver Rtp Receiver Track Track Rtp Listener

Sctp Transport Data Channel Sctp Transport Data Channel Data Channel Ice Transport Controller Quic Transport Quic Stream Quic Stream Quic Transport Quic Stream Quic Stream

Source: http://draft.ortc.org/

Quic Datagrams Quic Datagrams

WebRTC 1.0 API

PeerConnection

RTPSender

PeerConnection

JavaScript Application (Sender) JavaScript Application (Receiver)

Track

Track RTPSender

DTLS Transport ICE Transport Internet

ICE Transport DTLS Transport

RTP Receiver

RTP Receiver

Track

Track

SDP SDP

Track

Data Channel

SCTP Transport SCTP Transport

Track

Data Channel

WebRTC 1.0 CR to ORTC

PeerConnection RTPSender PeerConnection

JavaScript Application (Sender) JavaScript Application (Receiver)

Track Track RTPSender DTLS Transport ICE Transport

Internet

ICE Transport DTLS Transport RTP Receiver RTP Receiver Track Track RTPSender

JavaScript Application (Sender) JavaScript Application (Receiver)

Track Track RTPSender DTLS Transport ICE Transport

Internet

DTLS Transport RTP Receiver RTP Receiver Track Track

SDP SDP Objects Object s

ICE Transport

Can't control directly

Control directly

Questions

• Why are the QuicTransport and QuicStream objects shown as

included only within the ORTC model?

• Why might it be useful for QUIC to support unreliable transport

(QUIC datagrams) as well as reliable transport (QUIC streams)

• What are the potential advantages of transporting media as well as

data over QUIC?

WebTransport + WebCodecs

• Support for gQUIC data exchange was provided within Chrome and Edge

Beta as an origin trial, based on the ORTC model.

• Subsequently, work has begun on a client-server protocol and API known

as “WebTransport”.

• WebTransport complements WebSockets: provides reliable transport via

QuicStreams as well as unreliable transport via QUIC datagrams.

• To enable transport of streaming media as well as realtime

communications over QUIC, work has begun on a WebCodecs API.

• https://discourse.wicg.io/t/webcodecs-proposal/3662
• The Origin Trial and other ongoing efforts relating to QUIC are described

in a video, available here:

• https://www.youtube.com/watch?v=964yH8GoD-I

Capture, Devices and Output

MediaStreams

• A MediaStream represents a stream of media content, consisting of

several MediaStreamTracks.

• MediaStreamTracks can be added to a MediaStream using addTrack()
• MediaStreamTracks can be removed from a MediaStream using removeTrack()
• getTracks(), getAudioTracks() and getVideoTracks() can be used to obtain the

MediaStreamTracks within a MediaStream

• A MediaStream object can be obtained by:
• Using the constructor,
• Calling getUserMedia() to capture from a device,
• Calling getDisplayMedia() to capture from the screen,
• Calling captureStream() to capture from an <audio>, <video> or <canvas> element.
• Blog: https://developers.google.com/web/updates/2016/10/capture-stream
• A MediaStream object has a unique id.

MediaStream Interface

https://w3c.github.io/mediacapture-main/

Sources and MediaStreamTracks

• A source provides a MediaStreamTrack
• Example sources: physical webcam, microphone, screen capture, etc.
• A MediaStreamTrack has properties:
• id (unique identifier)
• kind (audio, video, depth, etc.)
• label that identifies the track source
• enabled that indicates whether the track can render audio or video. Setting enabled

to false can be used for muting (stopping audio or sending black video)

• muted If one of the SSRCs received by an RTCRtpReceiver is removed due to receipt
• f a BYE or timeout, the MediaStreamTrack is set to muted.

Properties of a MediaStreamTrack (cont’d)

• contentHint: a string that provides a hint about what the track contains
• Necessary in the absence of performant algorithms to auto-detect the

content type and optimize accordingly.

• Example: optimization of encoder settings (e.g. “text” hint turns on screen

capture tools in AV1)

• Specification: https://w3c.github.io/mst-content-hint/
• Demo:

https://webrtc.github.io/samples/src/content/capture/video-contenthint/

Content-Hints: Audio Values

Content-Hints: Video Values

MediaStreamTrack Interface

https://w3c.github.io/mediacapture-main/

Constraints, Capabilities and Settings

• constraints limit the operating modes that a source can use when

providing a MediaStreamTrack.

• navigator.mediaDevices.getSupportedConstraints() returns a list of supported

constraints.

• track.getCapabilities() returns a list of each supported constraint and the values
• r ranges of values that are supported.
• track.applyConstraints() specifies the values or range of values that are desired

for the constraints.

• track.getConstraints() returns the constraints most recently provided as an

argument to applyConstraints(). This may not represent the actual state of the track.

• MDN entry:

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints

• Settings represent the current configuration.
• track.getSettings() returns the track’s current configuration.

Note: remote tracks have settings but few operative constraints.

How Do We Obtain Streams/Tracks and Render Them?

• Capture
• Screen Capture: https://w3c.github.io/mediacapture-screen-share/
• Media Capture and Streams: https://w3c.github.io/mediacapture-main/
• Media Capture from DOM Element:

https://w3c.github.io/mediacapture-fromelement/

• Recording
• MediaStream Recording: https://w3c.github.io/mediacapture-record/
• Output
• Audio Output: https://w3c.github.io/mediacapture-output/

Screen Capture API

Supported browsers

Example Usage

var constraints = { video: true }; function handleSuccess(stream) { video.srcObject = stream; } function handleError(error) { console.log('navigator.mediaDevices.getDisplayMedia error: ', error); } navigator.mediaDevices.getDisplayMedia(constraints). then(handleSuccess).catch(handleError);

For More Information

• Blog:

○ https://blogs.windows.com/msedgedev/2018/05/02/bringing-screen

• capture-to-microsoft-edge-media-capture-api/
• Demos
• a. https://webrtc.internaut.com/iit-2019/screen-capture/
• b. https://webrtc.github.io/samples/src/content/getusermedia/getdispla

ymedia/

Questions

• Why does the Screen Capture API not permit the application to

influence what is captured?

• Why can’t the application determine what window is being

captured?

• Why might it make sense to set the contentHint property for the

video MediaStreamTrack in a Screen Capture?

• How can the application determine what to set it to?
• Why is the user always prompted for permission?
• Does it make sense to capture audio as well as video?
• What are some of the risks of audio capture?
• What behavior would you expect from occluded windows?

Minimized windows?

enumerateDevices API

Supported browsers

navigator.mediaDevices.enumerateDevices().then(gotDevices).catch(handleError);

getSupportedConstraints & getUserMedia APIs

Supported browsers

constraints = { audio: true, video: { width: { min: 1024, ideal: 1280, max: 1920 }, height: { min: 776, ideal: 720, max: 1080 } } };

Constraints

Audio Output Device API

Supported browsers

Questions

• Why do these APIs require SecureContext?
• Why do getUserMedia() implementations allow persistent

permissions?

• Why is the user required to provide permission before

enumerateDevices can return label and deviceId?

• What happens if getUserMedia() is called on a device with no

microphone or camera?

• What affect will this have on the ability of an application to

use setSinkId?

Demo

https://webrtc.github.io/samples/src/content/devices/input-output/

Questions?

The Evolution of WebRTC

Basics of WebRTC 1.0

• The WebRTC API is based on the semantics of SDP Offer/Answer as

defined in RFC 3264.

• Offer
• An Offerer configures the user-agent (such as by adding streams,

tracks or transceivers) and then calls createOffer() to generate an SDP Offer.

• The Offerer then calls setLocalDescription(Offer) to prepare the

user-agent based on the Offer.

• New: setLocalDescription() implicitly calls createOffer and

passes the resulting Offer as an implicit argument

• In the WebRTC 1.0 Candidate Recommendation (CR), the SDP

cannot be changed between createOffer and setLocalDescription, but initially, the SDP could be “munged”.

• The Offerer then sends the Offer to the Answerer.

Basics of WebRTC 1.0 (cont’d)

• Answer
• The Answerer calls setRemoteDescription(Offer), then configures

itself (by adding streams, tracks or transceivers)

• The Answerer calls createAnswer() to prepare an Answer and

setRemoteDescription(Answer).

• The Answerer sends the Answer to the Offerer.
• The Offerer calls setRemoteDescription(Answer).

Code Example

https://w3c.github.io/webrtc-pc/#simple-peer-to-peer-example

• Since its conception in 2013, the WebRTC API has gone through three

major iterations:

• addStream and removeStream (widely implemented legacy API)
• Implemented without a realtime object model, SDP only
• addTrack and removeTrack (implemented)
• Typically implemented with (incompatible) SDP and

RtpSender/RtpReceiver objects.

• addTransceiver (Favored WebRTC 1.0 CR API, now available in all

browsers)

• Requires support for “Unified Plan” SDP
• Full support for the object model
• Blog by Jan-Ivar Bruaroey of Mozilla:
• https://blog.mozilla.org/webrtc/the-evolution-of-webrtc/

The Evolution of WebRTC 1.0

• Audio and Video MediaStreamTracks are combined into MediaStreams, with

MediaStream.id (the “msid-id”) communicated to the remote peer along with MediaStreamTrack.id (the “msid-appData”) in SDP.

• Pros
• Allowed the remote peer’s MediaStream.id to align with the msid of the local peer.
• Aligned well with APIs that produce MediaStreams (such as getUserMedia())
• Cons
• MediaStream.id from multiple PeerConnections can collide.
• Adding a MediaStreamTrack to a MediaStream did not automatically result in it being

sent to the remote peer, since that is controlled by Offer/Answer.

• Replacing a MediaStreamTrack (such as by changing the camera from front to back)

requires re-negotiation since it results in a change to the SDP.

• More logical for encoders and decoders to operate on tracks, not streams.
• AddStream/RemoveStream removed from WebRTC 1.0 specification

AddStream/RemoveStream Concepts

# First MediaStream - id is 4701... m=audio 56500 UDP/TLS/RTP/SAVPF 96 0 8 97 98 a=msid:47017fee-b6c1-4162-929c-a25110252400 f83006c5-a0ff-4e0a-9ed9-d3e6747be7d9 m=video 56502 UDP/TLS/RTP/SAVPF 100 101 a=msid:47017fee-b6c1-4162-929c-a25110252400 b47bdb4a-5db8-49b5-bcdc-e0c9a23172e0 # Second MediaStream - id is 6131.... m=audio 56503 UDP/TLS/RTP/SAVPF 96 0 8 97 98 a=msid:61317484-2ed4-49d7-9eb7-1414322a7aae b94006c5-cade-4e0a-9ed9-d3e6747be7d9 m=video 56504 UDP/TLS/RTP/SAVPF 100 101 a=msid:61317484-2ed4-49d7-9eb7-1414322a7aae f30bdb4a-1497-49b5-3198-e0c9a23172e0

msid SDP (draft-ietf-mmusic-msid)

Why is there an object model in WebRTC 1.0?

• Need a way to tweak parameters on individual tracks sent over the wire

○ Bitrate ○ Framerate ○ Direction (sendonly/recvonly etc.)

• Existing control surfaces insufficient:

○ createOffer params - not per-track ○ AddStream params - not modifiable post-add ○ MST constraints - affects raw media, not encoding

• Still in the spec, implemented by all WebRTC browers
• MediaStreamTracks are encoded by RTCRtpSender objects, and decoded by

RTCRtpReceiver objects.

• RTCRtpReceiver’s track has a unique (and permanent) id, unrelated to

RTCRtpSender.track (which can change via replaceTrack).

• An RTCRtpSender with a null track does not send (not the same as track.enabled false!)
• MediaStreamTracks are configured by calls to addTrack/removeTrack/replaceTrack,
• ntrack fired due to remote tracks
• Remote projection of streams constructed entirely from addTrack, no side effects
• RTCRtpSender.replaceTrack() can be used to change cameras without SDP

re-negotiation.

• Aspects of the encoder that are within the “negotiated SDP envelope” (such as

maximum bitrate, maximum framerate, active/inactive, etc.) can be configured without SDP negotiation, by calling RTCRtpSender.setParameters().

AddTrack/removeTrack/onTrack Concepts

• SDP m-lines are inherently bi-directional, implying a pairing of RTCRtpReceiver and

RTCRtpSender objects.

• addTrack returns an RTCRtpSender; onTrack event returns an RTCRtpReceiver on the

remote peer.

• But which RTCRtpReceiver and RTCRtpSender objects go together?
• Using setParameters() with addTrack is ill-defined.
• addTransceiver used to set up simulcast layers to send before calling createOffer and

setLocalDescription.

• Calling setParameters() to change the number of layers *after* SLD throws!
• addTrack() adds a track and sets the “negotiationneeded” flag.
• removeTrack() does not undo addTrack()!
• Changes sender’s direction (“sendrecv” -> “recvonly”, “sendonly” -> “inactive”)
• Sets sender.track to null
• Causes remote track to be muted rather than ended.

AddTrack/onTrack Issues

addTransceiver Concepts

• Now supported in all browsers.
• addTransceiver() creates a RTCRtpTransceiver object which includes

RTCRtpSender/RTCRtpReceiver objects.

• Allows RTCRtpSender to be initialized with sendEncodings, establishing the

“encoding envelope” for simulcast (number of streams, order of encodings).

• setParameters() cannot change “encoding envelope” though it can set individual

streams to active/inactive.

• RTCRtpTransceiver has an m-line identifier (mid) attribute that identifies the

transceiver uniquely on both the local and remote peer.

• Track.id rarely correlates due to RTCRtpSender/RTCRtpReceiver model.
• Implies that each m-line corresponds to one and only one

RTCRtpReceiver/Sender pair. Only true in “Unified Plan” SDP.

• Blog by Jan-Ivar Bruaroey of Mozilla:
• https://blog.mozilla.org/webrtc/rtcrtptransceiver-explored/

• Since SDP is bi-directional, RtpTransceiver pairs the RtpSender and

RtpReceiver objects sharing an m-line.

• setDirection enables “hold” scenarios (“sendrecv”, “sendonly”, “recvonly”
• r “inactive”)
• Note: addTrack () returns an RtpSender and track event returns an

RtpReceiver.

• RtpSender and RtpReceiver may not both exist at a given time.
• RtpTransceiver objects “vended” by pc.addTransceiver().
• RtpReceiver object can be accessed via transceiver.receiver, RtpSender object

via transceiver.sender.

• pc.getTransceivers() returns the set of transceivers.
• pc.getSenders() returns an RTCPeerConnection’s “set of senders”
• pc.getReceivers() returns an RTCPeerConnection’s “set of receivers”.

RtpTransceiver Object

RTCPeerConnection Interface Extensions

Concepts

• RtpTransceiver (combination of sender and receiver) “vended” by addTransceiver()
• RtpSender “vended” by addTrack (addStream deprecated)
• RtpReceiver returned by a track event (see next slide) (onaddstream deprecated)
• RtpTransceiver(s), RtpSender(s), RtpReceiver(s) can be retrieved via getTransceivers,

getSenders, getReceivers

Transceiver creation options (RtpTransceiverInit)

Concepts

• direction attribute useful in “hold” scenarios (can be changed via

transceiver.setDirection()

• sendEncodings useful in advanced video scenarios
• Examples: simulcast, bandwidth limitation (more later)

Track Event

Concepts

• event.receiver provides the RtpReceiver.
• event.track provides the remote track (same as receiver.track)
• event.streams provides the streams that the track is part of
• Specification unclear whether this is always present or not.
• event.transceiver provides the transceiver (there is no receiver.transceiver attribute).

From Jan-Ivar’s Blog

Questions?

Perfect Negotiation

What is “Perfect Negotiation”?

What if you could add and remove media from a WebRTC peer connection without having to worry about state, glare (signaling collisions), role (what side you’re on), or the condition of the connection?

• Peer A: pc.addTrack(track, stream)
• Peer B: pc.ontrack = ({streams}) => video.srcObject =

streams[0];

• Negotiation, glare handling/rollback, object creation all happens

automatically. Blog by Jan-Ivar Bruaroey of Mozilla:

https://blog.mozilla.org/webrtc/perfect-negotiation-in-webrtc/

What is “Glare”?

• “Glare” is a situation where two peers send an Offer to each
• ther.
• WebRTC does not define which peer gets to be the Offerer and

which is the Answerer.

• However, the API does provide a way for the “polite” peer to

return to the state prior to sending an Offer so that it can accept an Offer instead. This is known as “Rollback”.

• Rollback is implemented in Firefox but is work-in-progress in

Chrome and Edge Beta.

Premise: Perfect negotiation (Source: Jan-Ivar)

50

pc.addTrack(track, stream) pc.removeTrack(sender) RTCPeerConnection pc.addTransceiver(kind) pc.restartIce() pc.createDataChannel(name) tc.direction = “sendrecv” tc.sender.setStreams(streams) tc.stop() RTCRtpTransceiver negotiationneeded icecandidate High-level application methods Low-level signaling methods pc.createOffer() pc.createAnswer() tc.reject() pc.setLocalDescription() pc.setRemoteDescription() pc.addIceCandidate(candidate) signaling channel

“Perfect Negotiation”: A Status Report

• Required work items:

○ Overall Chromium “Intent to Implement & Ship” statement: ○ https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/OqPfCpC5RYU

• Support for negotiationneeded

○ In Firefox, Safari, Chrome 75+ and Edge beta.

• Rollback support (to handle glare)

○ Already supported in Firefox. ○ In development in Chrome and Edge Beta:

https://www.chromestatus.com/feature/5079977739419648

○ Note: interaction of rollback and the object model still under investigation.

• API cleanup (race conditions and side effects)

○ pc.restartIce() method, https://www.chromestatus.com/feature/5723155500892160 ○ setLocalDescription() without arguments implicitly calls createOffer() or createAnswer(), https://www.chromestatus.com/feature/6253672042332160

○ transceiver.stop, https://www.chromestatus.com/feature/5410592384876544

Demo: Perfect Negotiation by Jan-Ivar Bruaroey

https://blog.mozilla.org/webrtc/perfect-negotiation-in-webrtc/

Note: Currently, this demo only works correctly in Firefox.

Questions?

Brief Break (10 minutes)

Introduction to the WebRTC/ORTC Object Model

Data Channel

Rtp Sender Track

ORTC Object Model

Application Application Track Rtp Sender

Dtls Transport Ice Transport Ice Gatherer Ice Transport Controller Ice Gatherer Ice Transport Dtls Transport

Rtp Receiver Rtp Receiver Track Track Rtp Listener

Sctp Transport Data Channel Sctp Transport Data Channel Data Channel Ice Transport Controller Quic Transport Quic Stream Quic Stream Quic Transport Quic Stream Quic Stream

Source: http://draft.ortc.org/

Quic Datagrams Quic Datagrams

WebRTC 1.0 CR API (addTransceiver) ORTC

ICE DTLS RTP SCTP

PeerConnection

ICE DTLS RTP SCTP

Application Same objects as WebRTC Full direct control PeerConnection optional (via JS lib) Application Objects all read-only Indirectly controlled via PeerConnection Some direct control

WebRTC 1.0 API

PeerConnection

RTPSender

PeerConnection

JavaScript Application (Sender) JavaScript Application (Receiver)

Track

Track RTPSender

DTLS Transport ICE Transport Internet

ICE Transport DTLS Transport

RTP Receiver

RTP Receiver

Track

Track

SDP SDP

Track

Data Channel

SCTP Transport SCTP Transport

Track

Data Channel

WebRTC 1.0 Object Model

• RtpSender* (Section 5.2)
• RtpReceiver (Section 5.3)
• RtpTransceiver (Section 5.4)
• DtlsTransport (Section 5.5)
• IceTransport (Section 5.6)
• SctpTransport (Section 6.1.1)
• DataChannel (Section 6.2)
• DTMFSender (Section 7.2)

* all objects are prefixed with “RTC”

RtpTransceiver Interface

Concepts

• Each transceiver corresponds to an SDP m-line (mid)
• Transceivers always have sender and receiver attributes (not nullable)
• Transceivers are stop()’d as a unit (e.g. no sender.stop or receiver.stop)
• Transceiver direction set via setDirection()
• Codec preferences can be changed without SDP negotiation via setCodecPreferences().

• Many senders / receivers can be associated to a single

DtlsTransport (BUNDLE)

• Sender takes a single MediaStreamTrack
• Receiver emits a single MediaStreamTrack
• Media "kind" is typically "audio" or "video“ ( but could also

be “depth”)

• Capabilities describe what objects can be configured to do.
• Parameters describe how to encode/decode media on the

wire.

RTCRtpSender / RTCRtpReceiver

RtpSender Interface

Differences from ORTC

• getParameters() method used to retrieve parameters. No equivalent in ORTC (application

can store parameters passed to send(parameters)).

• setParameters method used to set parameters.
• replaceTrack() method instead of setTrack().
• No setTransport() method.
• No stop() method (can call transceiver.stop()).

RtpReceiver Interface

Differences from ORTC

• getParameters() method used to retrieve parameters. No equivalent in

ORTC (application can store parameters passed to receive(parameters)).

• No setTransport() method.
• No stop() method (can call transceiver.stop()).

RtpCapabilities

• Indicates what codecs, header extensions and FEC

mechanisms are supported by RtpSenders and RtpReceivers.

• For each codec, information is provided, including

name/mimeType, clockRate, maxptime/ptime/numChannels (for audio), rtcpFeedback, etc.

• Parameters and options dictionaries provided for codec-specific

capabilities.

• codecs attribute includes more than just media codecs
• RED, Retransmission (RTX), DTMF, CN and FEC mechanisms (e.g.

“ulpfec”, “flexfec”, etc.) are included as well.

• One RTX entry for each codec that can be retransmitted.

RTCRtpCapabilities

Differences from ORTC

• FEC mechanisms not included in capabilities.
• Codec parameters included in sdpFmtpLine

Capabilities Exchange (ORTC only)

• RTCRtpCapabilities designed for “capabilities exchange”
• Peers exchange sender/receiver capabilities (including preferred

Payload Types)

• Capabilities used to compute RTCRtpParameters passed to

send(parameters) and receive(parameters)

• Intersection of codecs, header extensions, feedback messages can

be computed without specific knowledge

• sending codecs, header extensions and feedback messages computed from

intersection of local sender capabilities and remote receiver capabilities.

• receiving codecs, header extensions and feedback messages computed from

intersection of local receiver capabilities and remote sender capabilities.

Capabilities Exchange (cont’d)

• Determining codec parameters is trickier.
• Several important sender codec parameters can be

determined from receiver codec capabilities. Examples:

• Opus: receiver useinbandfec capability copied to sender useinbandfec

parameter

• H.264/AVC: receiver profile-level-id capability copied to sender

profile-level-id parameter

• VP8/VP9: receiver max-fr/max-fs capability is copied to sender

max-fr/max-fs parameter

• Question 1: In what situations does “capabilities exchange” signaling make

sense?

Class Poll

RTCRtpSend/ReceiveParameters

• Indicates what codecs, header extensions, encodings, rtcp

settings, etc. are to be used for sending and receiving.

• Codec parameters resemble codec capabilities.
• Parameters dictionaries are provided for codec-specific settings.
• codecs attribute includes more than just media codecs
• RED, Retransmission (RTX), DTMF, CN and FEC mechanisms (e.g.

“ulpfec”, “flexfec”, etc.) are included as well.

• An RTX entry is provided for each codec that supports retransmission.

RTCRtpParameters

RTCRtpCodecParameters

Differences from ORTC

• Codec parameters included in sdpFmtpLine.

WebRTC 1.0 Encoding/Decoding Parameters

RTCDtlsTransport

• (ORTC-only) Supports forking via RTCCertificate interface (to enable

forked DTLS transports to reuse the same certificate/fingerprint)

• Derives SRTP keys via DTLS/SRTP exchange
• Encrypts/decrypts data channel packets
• Associated to a single RTCIceTransport
• Sends/receives packets over virtual RTCIceTransport circuit path

from local to remote party

• Requires fingerprint validation of DTLS certificate to prevent

man-in-the-middle attacks

RTCDtlsTransport Interface

Differences from ORTC

• No getLocalParameters, getRemoteParameters, start or

stop methods (handled in SDP)

RTCIceTransport

• (ORTC-only) Associated to a single local IceGatherer.
• Multiple IceTransport objects can share an IceGatherer (forking)
• Sends ICE connectivity tests to test communication paths between a

local and remote peer

• Responds to incoming ICE connectivity checks with the specified

remote username fragment

• Forms a virtual circuit over which DTLS and SRTP media packets can

flow

RTCIceTransport Interface

Differences from ORTC

• No forking support
• Combines IceGatherer and IceTransport functionality
• nselectedcandidatepairchange versus onpairchange
• pc.addIceCandidate versus IceTransport.addRemoteCandidate

WebRTC 1.0 RTCIceTransportState Machine (non-normative)

Differences from ORTC

• No transition from “failed” to “checking” (“failed” state is terminal)
• “disconnected” state can be reached via transient connectivity loss (not just consent failure)
• No transition from “completed” to “connected” (connectivity loss/consent failure transitions to disconnected)
• Spec is unclear how “completed” state is reached (requires “end-of-candidates” indication for each IceTransport)

RTCDataChannel

• Represents a bi-directional data channel between two peers.
• Enables sending data over an RTCDataTransport (ORTC) or

RTCSctpTransport (WebRTC 1.0).

SctpTransport

• Represents the SCTP association between two peers.
• Enables sending data over an RTCSctpTransport (WebRTC 1.0 or

ORTC).

Differences from ORTC

• pc.sctp versus DataChannel.transport (ORTC)
• No sctp.getCapabilities or sctp.stop

SctpTransport Interface

Differences from ORTC

• pc.sctp versus DataChannel.transport (ORTC)
• No sctp.getCapabilities or sctp.stop

RTCQuicTransport

• New interface to support QUIC transport under development in the

IETF QUIC WG.

• Can be used for transport of data or media via QuicStreams or

QuicDatagrams.

• Associated to a single RTCIceTransport
• Sends/receives packets over virtual RTCIceTransport circuit path

from local to remote party

• Requires fingerprint validation of QUIC certificate to prevent

man-in-the-middle attacks.

RTCQuicStream

• New Interface to support QUIC streams.
• Each RTCQuicStream is associated to a single RTCQuicTransport
• Streams added to an RTCQuicTransport by calling

createStream(parameters) method.

• Permits creation of bi-directional or uni-directional (send-only) streams
• When the local peer calls createStream(), an RTCQuicStream

appears on the remote peer.

WebRTC/ORTC Object Model *

• Mandatory to implement in WebRTC only:
• RtpTransceiver
• Mandatory to implement in ORTC only:
• IceGatherer
• Mandatory to implement in both WebRTC and ORTC:
• IceTransport
• DtlsTransport
• RtpSender / RtpReceiver
• DataChannel
• SctpTransport
• Optional to implement in both WebRTC and ORTC:
• QuicTransport / QuicStream
• Optional to implement in ORTC
• IceTransportController
• RtpListener

* all objects are prefixed with “RTC” in the WebRTC and ORTC API specifications

WebRTC 1.0 Specification Additions

PeerConnection .getSenders() .getReceivers() .getTransceivers() .addTrack() .removeTrack() .addTransceiver() .sctp ... RtpSender .track .transport .getCapabilities() .getParameters() .setParameters(params) .replaceTrack(track) .setStreams() ... RtpReceiver .track .transport .getCapabilities() .getParameters() .getContributingSources() .getSynchronizationSources() .getStats() ... DtlsTransport .iceTransport .state .getRemoteCertificates() .onstatechange .onerror ... IceTransport .role .component .state .gatheringState .getLocalParameters(), .getRemoteParameters(), .getLocalCandidates() .getRemoteCandidates() .getSelectedCandidatePair() .onstatechange .ongatherinstatechange .onselectedcandidatepairchange ... SctpTransport .transport .state .maxMesageSize .maxChannels .onstatechange

RtpParameters

.headerExtensions .rtcp .codecs RtpSendParameters: RtpParameters .encodings .degradationPreference .priority RtpReceiveParameters: RtpParameters .encodings RtpCodingParameters .rid RtpEncodingParameters .rid .codecPayloadType .dtx .active .ptime .maxBitrate .maxFramerate .scaleResolutionDownBy IceParameters (read only) .usernameFragment .password

Source: https://w3c.github.io/webrtc-pc/

• "Warm up" media path while the getting a track and ringing
• Change the send codec (without SDP munging)
• Change the camera source instantly (front to back)
• Enable/disable sending of media instantly (without signalling)
• Set a maximum bitrate or maximum framerate
• Obtain detailed status of individual ICE and DTLS transports
• Send simulcast
• Receive simulcast (optional)

What you can do with WebRTC 1.0 objects

Example: Warmup

// Webrtc var audio = pc.addTransceiver("audio"); var video = pc.addTransceiver("video"); renderTrack(video.receiver.track); renderTrack(audio.receiver.track); // ... do getUserMedia and offer/answer // ... wait for "real answer" audio.sender.replaceTrack(audioTrack); video.sender.replaceTrack(videoTrack); // ORTC var audioSender = new RTCRtpSender(...); var videoSender = new RTCRtpSender(...); var audioReceiver = new RTCRtpReceiver(...); var videoReceiver = new RTCRtpReceiver(...); renderTrack(video.receiver.track); renderTrack(audio.receiver.track); // ... do getUserMedia and call signalling // ... wait for "real answer" videoSender.setTrack(videoTrack); videoSender.send(...); audioSender.setTrack(audioTrack); audioSender.send(...);

Example: Change camera

// Webrtc var transceiver = pc.addTransceiver(video1); var sender = transceiver.sender; sender.replaceTrack(video2); sender.replaceTrack(video1); // ORTC var videoSender = new RTCRtpSender(...); videoSender.setTrack(video2); videoSender.setTrack(video1);

Example: Change send codecs

// Webrtc var transceiver = pc.addTransceiver(...); var sender = transceiver.sender; // After every call to // setLocalDescription and // setRemoteDescription: var p = sender.getParameters(); p.codecs = reorderCodecs(p.codecs); sender.setParameters(p); // ORTC var sender = new RTCRtpSender(...); // Only once, but choose the PTs var codecs = reorderCodecs( sender.getCapabilities().codecs); sender.send({codecs: codecs, ...});

Example: Set maximum bitrate

// Webrtc var transceiver = pc.addTransceiver(...); var sender = transceiver.sender; var p = sender.getParameters(); // bps p.encodings[0].maxBitrate = 1000000; sender.setParameters(p); // ORTC var sender = new RTCRtpSender(...); sender.send({ encodings: [{maxBitrate: 1000000, ...}] });

Example: Enable/disable media

// Webrtc var transceiver = pc.addTransceiver(...); var sender = transceiver.sender; var p = sender.getParameters(); p.encodings[0].active = false; sender.setParameters(p); p.encodings[0].active = true; sender.setParameters(p); // ORTC var sender = new RTCRtpSender(...); sender.send({ encodings: [{active: false, ...}] }); sender.send({ encodings: [{active: true, ...}] });

Implementation Status

• Web-platform-tests dashboard “does not contain useful metrics for evaluation or

comparison of web platform features”

• Web confluence project:

○

Looks at properties and methods exposed by browsers:

○

https://web-confluence.appspot.com/#!/

○

Caveat: no guarantee that a widely-supported API is interoperable in its details, or will remain part of the web platform.

○

Tool that extracts data from the confluence tracker: https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc

• Overall status:

○

Fewer features with no implementations, compared with TPAC 2018.

○

Improved object model support

■

TPAC 2018: Current edge the only browser implementing DtlsTransport and IceTransport.

91

Implementation Status (cont’d)

92

Implementation Status (cont’d)

93

Implementation Status (cont’d)

94

Implementation Status (cont’d)

95

Demo

• Object Inventory and Capability dumper:
• https://webrtc.internaut.com/iit-2019/cap-dumper/
• Tells you what objects a browser supports
• Dumps RtpSender/RtpReceiver.getCapabilities(kind) for “audio”

and “video”

Interoperability Status

• Web Platform Test (WPT) status: https://wpt.fyi/webrtc
• More green, but still some yellow, orange and red.

○ 1+ implementations of all objects: RtpSender/Receiver, SctpTransport, DtlsTransport, IceTransport ○ Still some false negatives due to dependencies.

• No WPT tests for simulcast

○ “Simulcast playground” demonstrates the value of a loopback test ○ “Spec” approach doesn’t work with unified plan on receiving end

97

WPT Status: Orange is the new Pink

98

WPT Status (cont’d)

99

WPT Status (cont’d)

100

Questions?

Advanced Video: Simulcast & Scalable Video Coding (SVC)

https://w3c.github.io/webrtc-svc/

Simulcast

• Simulcast involves sending (and possibly receiving) multiple streams

that differ in characteristics such as resolution or frame rate.

• Typical scenario:
• Browser sends multiple streams (e.g. a high resolution and a low resolution

stream) to a Selective Forwarding Unit (SFU).

• SFU selects which stream to forward to each conference participant.
• Mobile device receives the low resolution stream
• PC receives the high resolution stream
• Both WebRTC 1.0 and ORTC object models support sending multiple

streams.

• ORTC also supports receiving multiple streams (and outputting a single

track).

Typical Simulcast Scenario

SFU

Multiple encodings from a single source Single encoding

WebRTC API Simulcast Examples

Source: https://w3c.github.io/webrtc-pc/#rtcrtpencodingspatialsim-example*

Demo: Simulcast-Playground by Fippo

https://github.com/fippo/simulcast-playground

Scalable Video Coding

• Codecs implemented in WebRTC support scalable video coding (SVC):
• VP8: temporal scalability
• VP9: temporal and spatial scalability
• H.264/SVC: temporal and spatial scalability
• AV1: temporal and spatial scalability
• SVC now mainstream
• Utilized in many products: Google (Hangouts), Microsoft (Teams), Vidyo, Polycom,

Avaya/RADVISION, LifeSize, etc.

• ORTC makes it possible to enable SVC in the RtpSender
• Developer can specify the number of layers but typically just wants the

browser to “do the right thing” (send as many layers as conditions permit).

• As long as the decoder supports SVC, no need to configure the RtpReceiver

unless differential protection is used (e.g. applying FEC or RTX only to the base layer).

• WebRTC-SVC API enables support of SVC in WebRTC 1.0

AV1 Predefined Scalability Modes (Section 6.7.5)

Note: “S” modes represent simulcast.

https://aomediacodec.github.io/av1-spec/av1-spec.pdf

L1T2 Scalability Mode

L1T3 Scalability Mode

L2T1 Scalability Mode

L2T2 Scalability Mode

L2T3 Scalability Mode

S2T1 Scalability Mode

S2T2 Scalability Mode

S2T3 Scalability Mode

• Question 1: Who is still awake?
• Question 2: Do “S” scalability modes make sense when

transporting AV1/SVC over RTP? Why or why not?

• Question 3: What interoperatability issues might result?

Class Poll

ORTC API Examples: Temporal Scalability

Source: http://draft.ortc.org/#rtcrtpencodingtemporal-example*

• Can encodingId/dependencyEncodingId be used to describe any

scalability structure?

Questions

L3T2_KEY Scalability Mode

• Question 1: Why can’t L3T2_KEY be described via a dependency

structure?

• Question 2: How is L3T2_KEY different from L3T2 and S2T2?
• Question 3: What is the advantage of L3T2_KEY over L3T2?

Class Poll

WebRTC-SVC Capabilities Example

Source: https://w3c.github.io/webrtc-svc/#example-3

• In what situations is it important to indicate the scalability modes

supported by a Receiver?

Questions

WebRTC-SVC API Examples

Source: https://w3c.github.io/webrtc-svc/#example-1

WebRTC-SVC API Examples (cont’d)

Source: https://w3c.github.io/webrtc-svc/#example-2

Questions?

RTCPeerConnection Interface

RTCPeerConnection Interface (cont’d)