Standard approaches and a case study Corso di Sistemi Multimediali - - PowerPoint PPT Presentation

Standard approaches and a case study

1

Corso di Sistemi Multimediali A.A. 2010-11

• Prof. Antonio Picariello

Corso di Applicazioni Telematiche A.A. 2010-11

• Prof. Simon Pietro Romano

Università degli Studi di Napoli Federico II Facoltà di Ingegneria 9/6/2011

Outline

 Web Conferencing Introduction

 History and technologies

 IETF standardization efforts

 Milestones and ongoing works in the RAI area

 Centralized Conferencing, Media Control, Session Recording,

Via-browserconferencing, Telepresence Systems

 From theory to practice: the Meetecho platform

 A fully-fledged standard conferencing system made in

unina

2

A “Web Conferencing” definition

 The term “Conference” can be used to describe any meeting

• f people that “confer” about a certain topic

 “Web Conferencing” is used to conduct live meetings or

presentations over the Internet

3

Features

 VoIP (Voice over IP)  Live video  Text chat  Slide presentations  Whiteboard

with annotation

 Screen/desktop

sharing

 Application sharing  Recording  Polls and surveys

4

History

 Tele-Conferencing

 Conference calls

(Audio Tele-Conferencing)

 Video conferences

(Video Tele-Conferencing)

 IP-Conferencing

 Text Conferencing  Audio and Video Conferencing  Data Conferencing

5

Audio and Video Conferencing Evolution

 Analog Phone Lines (PSTN)

 Conference calls

 Three-way calling  Conference bridges

 Video Telephony

 Digital Telephony (ISDN)

 ITU-T H.320 umbrella recommendation

 Next-Generation Networks & VoIP technologies

6

VoIP protocols

 SIP – Session Initiation Protocol, RFC 3261

 Call signalling

 Session intitiation, managing and termination

 Codec negotiation

 SDP – Session Description Protocol, RFC 4566

 Session media stream description

 RTP – Real-time Transport Protocol, RFC3550

 Media streams transfer

7

Text Conferencing Evolution

 Asynchronous Meetings

 Posted text messages (not live)

 Message/Bulletin Boards  Forums  Network Newsgroups/Mailing lists

 Synchronous Meetings

 Live text communication

 Internet Relay Chat (IRC)  Web-based Chat (CGI/Java)  Instant Messaging (Skype/MSN/XMPP/etc.)

8

A B C A+B+C A+B

Typical scenarios

 From Point-to-Point Calls to Multipoint Calls

 Three-way calling  Coaching scenario

 Lecture-mode Conferences

 Presentation  Question & Answers session

 Ad-hoc and Reserved Conferences

 Conference-aware/-unaware participants  Manage conference/users/media/policies  Sidebars/Whispers

9

Client B Client C Client A M e d i a A M e d i a C Media B Server

A B B A A+B

Issues

 Call signalling

 Gateway functionality

 Control and Management

 Tone detection (DTMF)  Dedicated protocols

 Mixing and Transcoding

 Terminal capabilities  User media profiling

 Coaching scenario  Videoswitching

10

Client B Client A S I P H . 3 2 3 Server

Standardization Efforts

 Geographically dispersed businesses, the rising cost of energy, and the

development of NGN lead to the proliferation of proprietary conferencing solutions

 No standardization for many years

 Lack of interoperability  Platform dependency  Security issues  Cost  Market segmentation

 Standardization Bodies

 ITU (International Telecommunication Union)  3GPP (3rd Generation Partnership Project)  IETF (Internet Engineering Task Force)

11

Standardization Efforts: ITU

 Established to standardize and regulate international radio and

telecommunications

 International Standards referred to as “Recommendations”  ITU-T: Telecommunication Sector

 G: Transmission Systems and Media

 G.71x (Audio compression, mu-law and a-law)  G.72x (Audio compression, ADPCM)

 H: Audiovisual and Multimedia Systems

 H.320 (PSTN/ISDN, Telephone Systems)  H.323 (IP, Packet-based Communication Systems)

 T: Terminals for Telematic Services

 T.120 (Data Sharing Protocols)  T.140 (RTP Interactive Text)

12

Standardization Efforts: IETF

 Under the umbrella of the Internet Society  Develops and promotes Internet Standards  Deals in particular with standards of the TCP/IP suite  Organization

 Working Groups (WG)  Internet Drafts  Requests for Comments (RFC)  “Rough consensus, running code”

 We will focus on some RAI area activities...

13

IETF RAI area

 RAI – Real time Applications and Infrastructure  “Legacy” standards

 Standard VoIP protocols  SIPPING  XCON  MediaCtrl

 Brand-new standardization activities

 SIPREC  RTCWEB  CLUE

14

SIPPING Working Group

 Session Initiation Protocol Proposal INvestiGation  Documents the use of SIP for several applications related to

telephony and multimedia

 SIP Conferencing Models

15

Loosely-Coupled Conference Fully Distributed Multiparty Conference Tightly-Coupled Conference

Sip Conferencing Framework

Main actors:

• Participants
• Focus
• Mixer
• Policy Server
• Notification Server

XCON Working Group

 Centralized Conferencing (“X”CON)

 A star topology on the signalling plane

 Advanced conferencing features

 VoIP + video & data sharing

 “Signalling-agnostic”

 Not only SIP

 H.323, IAX, …

 Defines :

 Conference data model and lifecycle  A suite of client-server protocols to realize sophisticated

conferencing scenarios

16

Conference Object &Data Model

17

first join Template conference object (blueprint) Registered conference object Active conference object cloning creation last leave <conference-info> <sidebars-by-val> <sidebars-by-ref> <conference-description> <host-info> <floor-information> <conference-state> <users> delete delete

Conference object example

18

<inf nfo:c :con

• nfer

eren ence-in info entity="8773158"> <in info fo:co conf nfere renc nce-des escri ripti tion

• n>

<info:display-text>My Full-Media Conf</info:display-text> <info:conf-uris> <info:entry> <info:uri>xcon:8773158@meetecho.com</info:uri> <xcon:conference-password>3903/0</xcon:conference-password> </info:entry> </info:conf-uris> <info:available-media> <info:entry label="audioLabel"> <info:type>audio</info:type> </info:entry> <info:entry label="videoLabel"> <info:type>video</info:type> </info:entry> </info:available-media> </ </inf nfo:c :con

• nfer

eren ence-de descr cript ptio ion> <in info fo:co conf nfere renc nce-sta tate> e> <info:active>false</info:active> </ </inf nfo:c :con

• nfer

eren ence-st state te> <in info fo:us user ers> <info:user entity="13"> <info:display-text>alex</info:display-text> <info:endpoint entity="sip:alex@130.129.20.143:5060"/> </info:user> <info:user entity="14"> <info:display-text>user134</info:display-text> <info:endpoint entity="sip:user134@39.7.138.42:5080"/> </info:user> <xcon:join-handling>allow</xcon:join-handling> </ </inf nfo:u :use sers> <xc xcon

• n:fl

floo

• or-inf

nform rmat ation

• n>

<xcon:floor-request-handling>confirm</xcon:floor-request-handling> <xcon:conference-floor-policy> <xcon:floor id="11"> <xcon:media-label>audioLabel</xcon:media-label> <xcon:max-floor-users>8</xcon:max-floor-users> <xcon:moderator-id>13</xcon:moderator-id> </xcon:floor> <xcon:floor id="22"> <xcon:media-label>videoLabel</xcon:media-label> </xcon:floor> </xcon:conference-floor-policy> </ </xco con:flo loor-in infor

• rma

matio ion> </ </in info:

• :co

confe fere rence ce-info fo>



Conference ID



Title



Conference state



Password / PIN



Allowed join modality



Floor handling policy



Participants info



User ID



Nickname



Signalling URI 

Floor info



Floor ID



Associated media



Moderator ID



Max number of floor owners

XCON Protocols

 Signalling

 SIP, H.323, IAX2, et al.

 Floor Control

 BFCP

(Binary Floor Control Protocol)  Notification

 Xcon Event Package

 Conference Control

 CCMP

(Centralized Conferencing Manipulation Protocol)

 University of Napoli highly

active in this field

19

CCMP – Conference Control

 Conference CRUD Manipulation

 Creation

 From a client-provided model or

by cloning system blueprints

 Retrieval  Update

 User profiles, multimedia flow

features (audio volume, video layout, …),...

 Deletion

 XML- based

 Carrying datamodel fragments

 “Single verb HTTP + CCMP body”

transport solution

20

BFCP – Floor Control

 Aimed at coordinating access to a set of shared resources

 A “Floor” is a token, a temporary permission to access or

manipulate a specific shared resource or set of resources

 Standardized in RFC 4582

 Identifiers (Conferences/Floors/Users)  Floor Control Server  Floor Control Participant  Floor Chair

 Negotiation of BFCP connections within SIP/SDP

standardized in RFC 4583

 Only existing implementation to date: COMICS/Ericsson

21

BFCP in action

22

1) Floor Request 2) Notify 3) Chair decision 4) Decision 5) Floor Granted/Denied 6) Notify

Chair Decision

1 2 3 4 5 6

Floor Request Floor Granted Or Denied Decision Notify Notify

MediaCtrl Working Group

 Defines an architecture in which an Application

Server can control a Media Server

 A Media Server could be completely devoted to

perform onerous online/offline operations on media flows:

 Media Processing



Mixing/Transcoding



Playing/Recording



Storing/Retrieving



Detecting Tones (DTMF)



Interactive Voice Response (IVR)/VoiceXML



Text-to-Speech/Speech Recognition

 RTP Streams Manipulation

 Exploitable approach within conferencing

frameworks

23

Client B S I P S I P Application Server Media Server S I P Control Channel RTP R T P

SIPREC Working Group

 SIPREC - SIP-based RECording

 Designing a protocol based on SIP-extension and a metadata model to deal with

multimedia session recording  A recorded session can play an important role in different scenarios

 E-learning, minutes taking, …  Business analytics  Media processing (online/offline)  …

 Some media processing actions may require separation between different

media streams

 E.g.:



Word-spotting,



Speaker identification



Emotion-detection



…

24

SIPREC - Architecture

 Session Recording Client

(SRC)

 Logical entity able to

acquire the communication session to be recorded

 Sends the recording media

streams to the SRS  Session Recording Server

(SRS)

 Records the received session

 Recording Session (RS) is

made of:

 CS Multimedia Content  CS Metadata

25

SIPREC - Requirements

 “Active” recording

 The SRC proactively demands for registration  No passive recording / sniffing / wiretapping

 Session participants MUST be recording-aware

 Notification mechanisms are needed to inform participants

 Start, Pause, Resume and Stop controls  Define a protocol between SRC and SRS to manage the

recording session

26

SIPREC – Recording Metadata

 Information that describes recorded media and the

Communication Session (CS) to which they relate

 They have to allow for:

 Searching and filtering the archived session  Performing a “meaningful” playout

 e.g., with correct synchronization between media streams

27

Recording Metadata Example

<recording-metadata xmlns='...:siprec'> <recording id=""> </recording> <group id="" recording=""></group> <session id=""group=""></session> <participant id=""session=""></participant> <stream id=""session=""></stream> <extensiondata id="" parent=""> </extensiondata> </recording-metadata>

Recording Metatada Elements



Recording Session



Features: ID, recording type (selective, persistent), recording requestor (SRC or SRS), …



One or mode associated CSG 

Communication Session Group (CSG)



Identified by a unique ID



Group of one or more CS belonging to the same macro-session 

Communication Session (CS)



Features: ID, call termination reason, start time, end time, …



Can involve one or more Participant elements and one or more Media Stream elements 

Participant



Features: name, AoR, participant role, … 

Media Stream



Features: start time, stop time, codec, SDP m-line reference, .. 

Extension Data



Envisioned the optional presenxe of additional data not specified as part of SIPREC

29

RTCWEB Working Group

 Real Time Communication on the World Wide Web  Goal: standardizing basic components enabling interactive rich

communication using audio, video, collaboration, games, etc. between two peers' web-browsers

 Direct browser-to-browser communication

 Nowadays several different methods exists, but they are not interoperable since they

rely on incompatible browser plug-in platform  Collaboration with the W3C WEBRTC group :

 IETF  architecture and requirements  W3C  Javascript API development

 The idea: client-side applications, typically written in HTML and Javascript,

interacting with web browser through a yet-to-be-defined API

30

RTCWEB IETF issues

 Identification and definition of network-related

aspects

 Control protocols  Connection establishment and management  Connection-less transport  Selection of the most suitable encoders/decoders  Codec negotiation  New threats and security issues  Firewall and NAT traversal  Interoperation with legacy VoIP equipment, keeping the

balance with practicality of browser deployment

31

CLUE Working Group

32

 ControLling mUltiple streams for tElepresence

 Telepresence systems provide high definition, high quality

audio/video enabling a "being-there" /”immersive” experience

 Use of multiple

displays permitting life size image reproduction using multiple cameras, encoders, decoders, microphones and loudspeakers

CLUE issues

 specifications for SIP-based conferencing systems

to enable communication of information about media streams necessary to make reasonable decisions about transmitting, selecting, and rendering media streams:

 Spatial relationships of cameras, displays, microphones, and

loudspeakers

 Viewpoints - field of view/capture for

camera/microphone/display/loudspeaker

 Senders and intermediate devices use them to understand how best to

compose streams for receivers

 Receivers can know the characteristics of the received streams

 Usage of the stream (presentation, document camera output, …)  Aspect ratio of cameras and displays  Sources selection receiver-driven

33

Meetecho Platform

 Web conferencing and on-line collaboration tool

 Academic spin-off since Spring 2011  Developed by COMICS members and Unina students

 Main features:

 Scalable and Performing Client-Server Solution  IETF standards compliant (XCON, Media Ctrl)  Open-source  Java-based Desktop Client  Mobile device access support



Android



iPhone, iPad



BlackBerry



Symbian



Windows Phone  Web browser access support (Meetecho Weblite)

34

Meetecho capabilities

PSTN audio Voice over IP audio Live video Audio & Video Moderation Presentation delivery Desktop/screen sharing Text chat Presence File sharing Whiteboard Sharing Basic security Remote Control Session Recording Polls & Surveys Mobility support Web access support

Meetecho in action

Audio/Video controls panel

 Mirror On/Off  Pause Audio  Pause Video  Listen/Talk Volume  Selection of the desired

contribution

Moderated Scenarios

 Realized by means of BFCP  The conference chair decides to grant or not audio

and/or video floor to the requestor

 The Meetecho server allows only granted users to

contribute their media within the conference media mix

 Adaptive video layout depending on granted participant

number

Slide sharing

 Slides are converted in background to a series of images made available on a

web server

 All kind of presentation are supported

 Microsoft Office (ppt, pptx, doc, docx, xls, xlsx)  OpenOffice (odp)  Pdf  Slides previously exported in image

formats as (gif, jpg, png, ecc.)

 The presenter is the one who controls

slide progress

 Slides can be sent to the whiteboard

for online collaborative modifications

Whiteboard

 Shared between conference participants  Allows to

 Draw illustrative sketches  Modify slides  Insert clip-art  Save file modifications

Text Chat

 Based on XMPP (eXtensible Messaging and Presence

Protocol)

 Allows instant messaging during conference sessions

between conference participants

 Features

 Roster management  Presence info  Single file sharing  Conversation chronology

Polling

 Creating, answering and consulting real-time surveys

for conference participants

Desktop sharing

 Meetecho users can

 Show to other participants their desktop activities  Render desktop control to other participants  Set visualization quality

Video mixing (1)

 Up to 16 parallel video streams in a single conference  Mixing strategy:

 The mixer receives participant video contributions  Video streams are composed in a mosaico stream  The mixer sends to each participant the single mosaico

stream and not each single video contribution

 That allows for

 Constant load  Constant bandwidth consumption

Video mixing (2)

Server Meetecho Server Other tools overwhelm clients with as many flows as the contributing participants number… …Meetecho mixes video streams and sends always a single flow to each users Users Users User User

Meetecho codecs

 Supported audio codecs:

 GSM  G.711 u-law

 Supported video codecs:

 H.323 Q-CIF (176 x 144 pixel)  H.323 CIF (352x288 pixel)

46

Managing Scalability

 Centralized case

 A single focus for N users

 Distibuted case

 Cascaded Conferencing



Load-balancing users between more interoperating focuses



Each focus is seen as a participant by the other focus

 Orchestration of a set of XCON

“clouds”  Experimental campaign showed

performance benefits whit the increase

• f the participant number

47

Meetecho Session Recording

 Meetecho Session Metadata are encoded using SMIL

(Synchronized Multimedia Integration Language)

 Well known and established standard specification

 Recordings are available in a standard fashion

 Media timing – aware

 Enabling synchronized playout

 Allowing media stream separation

 A Recording Server is envisioned

 To receive all conference media streams and record them  To encode stream-related metadata info in the standard SMIL

format

48

Server-side recording

 Client-side recording may be realized by means of ordinary desktop

recording softwares

 Available tools: Adobe Captivate, Wink, Istanbul, Byzanz,…  Simple and fast  Conferencing platfom independent

 Client-side recording drawbacks:

 Only a single perspective is taken into account  Recorded media “flattened” to a single, very large video  Output files usually in proprietary formats



Flash, OGGTheora, …

 Meetecho server-side recording approach

 uses standard formats  is more suitable for multimedia session with several media and etherogeneous

devices involved

49

Recording audio & video

 Both Audio and video from

each participant are aggregated in the central mixer

 the mixer can record and

store them locally or on a remote storage  Transcoding, adaptation and

• ther operation may be

carried out on the streams

 The output is a single file

containing both audio and video (AVI file with specific codecs)

Mixer

UA-B UA-A

RTP RTP

XCON AS

Recorder

MEDIACTRL SIP SIP RTP UA-A (Rx + Tx) RTP UA-B (Rx + Tx)

Recording IM & events

 Chat contributions are

saved together with timestamps and info on the employed IM protocol

51

Recorder

UAC-B UAC-A

B (IRC) ‘10:11:26 – Hey C’ A (XMPP)

UAC-C

C (MSRP) ‘10:11:30 – Hello’ ‘10:11:24 – Hi!’ 10:11:24 <User C> Hi! 10:11:26 <User A> Hey C 10:11:30 <User B> Hello […] […]

 Textual event notification

may be treated as well

 Slide changes  Join /leave events  Poll announcement

Recording slide presentations

 Presentations are considered as simple slideshows of

static images

 Slide change events are triggered and notified via

XMPP, registered as text

 

Slides are registered as a series of images together with metadata info (associated events)

52

Recording whiteboarding

53

Whiteboard Server

UAC-B UAC-A

B (XMPP) 10:10:56 – circle A (XMPP)

UAC-C

C (XMPP) 10:12:30 – text 10:11:20 – line 10:10:56 <User A> circle 10:11:20 <User C> line 10:12:30 <User B> text […] […]

Putting It All Together: SMIL

 Standard XML-based markup language for describing

presentations involving heterogeneous media

 SMIL metadata file

 <head>

 Generic metadata and layout composition details

 <body>

 References to media to be included (URL)  Layout regions info  Timing info

54

Media metadata

 SMIL allows for both standard and non-standard media to

be included in a presentation

 Standard media streams have dedicated standard tags in

the media part of the <body>

 <video>

 the A/V Meetecho recording AVI file

 <textstream>

 IM, event notifications, converted in Real-Time Text docs

 <img>

 slides

 Non-standard media are included in the <ref> tag

 whiteboarding case

55

A Meetecho conf SMIL file

56

Playing out the recording (1)

57

Session database

Web server

SMIL- enabled player

• 1. START
• 7. SHOW
• 2. get conf45.smil

video text slides whiteboard

• 3. get audio/video
• 4. get RealText file
• 5. get slide images
• 6. get whiteboard packets

Playing out the recording (2)

 General approach:

1)

Retrieval of the SMIL file (HTTP or out-of-band mechanism)

2)

SMIL file is passed to a compliant media player preparying the presentation layout

3)

The player retrieves and reproduces each supported media

 SMIL players examples : RealPlayer, Ambulant,…

 Standard stream support

 Don’t play Meetecho whiteboarding…

  Meetcho players!

58

Playing out the recordings (3)

 3 options

1)

Watch it online by means of a JNLP- based custom player

2)

Watch it online with HTML5

3)

Download the registration and watch it offline (jar file)

59

 Session recordings available at:

https:\\ietf.conf.meetecho.com

60