Multimedia Communications Spring 2006-07 Advances in the Transport - - PowerPoint PPT Presentation

CS 584 / CMPE 584

Multimedia Communications

Spring 2006-07

Advances in the Transport Layer

(RTP)

Shahab Baqai LUMS

2

Outline Why RTP? RTP Features RTP Packet RTCP RTCP Reports Jitter Estimation

3

RTP

provide end-to-end network transport functions applications transmitting real-time multimedia data mode of transmission

– Unicast – multicast

data transport is monitored by a sister protocol RTCP

– allow monitoring of the data delivery – scalable to large multicast networks – provide minimal control and identification functionality.

RTP is proposed by IETF in the RFC 1889

– Audio/video profile in RFC 1890

4

Why RTP TCP cannot support the real time services

– Why??

UDP is used over IP as a better option

– Pros: multiplexing and check sum services – Cons: does not support retransmissions upon packet loss

RTP counters drawbacks of UDP for multimedia RTP may be used with suitable underlying network or transport protocols but with some expectations from lower level protocols

– Length indication – Framing mechanism – Possibility of packing multiple RTP packets into one network/transport packet

5

RTP Features

• RTP aids in

– payload type identification – sequencing of the out-of-order packets – Timely play out of the media data using timestamps

• RTCP

– monitors the quality of service – conveys information about the participants in an on-going session

• RTP is independent of the underlying protocol

– can work on any type of network like TCP/IP, ATM, frame relay etc.

• RTP supports multicast distribution if provided by the underlying network
• RTP sequence numbers can also be used to determine the proper

location of a packet

– E.g. sequence numbers can be used in video decoding, without necessarily decoding packets in sequence

6

RTP Does Not Provide any mechanism to ensure timely delivery, nor provides QoS guarantees

– Relies on lower layer services to do so

Does Not Guarantee delivery, nor prevents out-of-order delivery

– Does not assume that the underlying network is reliable nor that it delivers packets in sequence – Provides sequence numbers to determine the proper location of a packet

7

RTP RTP is a protocol framework

– As a protocol, it is deliberately not complete – RTP is tailored to applications through modifications, rather than by providing options

Complete specs for an application requires

– A profile specs

Defines

• Set of payload type codes
• Mapping of payload types to payload formats

– Payload format specs

8

RTP RTP uses:

– Application level framing – Integrated layer processing

Malleable to provide the information required by a particular application Often integrated into application processing rather than being implemented as a separate layer

9

RTP & OSI

PCMA audio MPEG2 video Application RTP RTP UDP UDP I P I P Network Ethernet Frame Relay Data Link Transport

• Which layer does RTP reside in?

10

RTP Encapsulation

• Video and audio payloads are sent separately
• Uses sequence number to synchronise audio and

video once received

IP Header UDP Header RTP Header RTP Video Payload IP Header UDP Header RTP Header RTP Audio Payload

11

RTP Packet RTP packet

– Fixed RTP packet header – List of contributing sources (possibly empty) – Payload data

RTP payload

– Data transported by RTP in a packet

E.g. audio samples, compressed video data, etc …

12

RTP Header

(V) Version; 2 bits (P) Padding; 1 bit. (X) Extension; 1 bit. (CC) CSRC Count; 4 bits. (M) Marker; 1 bit. (PT) Payload Type; 7 bits. Sequence Number; 16 bits. Time Stamp; 32 bits. SSRC; 32 bits. CSRC List;

SSRC CSRC identifiers …… Payload V=2 Sequence Number P X CC M PT Time Stamp

13

Header (cont)

• Version (V): The field is of length 2 bits

– indicates the current version of RTP. The current version of RTP is 2.0.

• Padding (P): This field is of length 1 bit.

– If P is set, the packet contains one or more additional padding octets at the end, which are not part of the payload. – Last octet of padding contains a count of how many padding octets are to be ignored – Padding is needed by some encryption algorithms, which require fixed block sizes, or for carrying several RTP packets in a lower-layer PDU.

• Extension (X): This field is of length 1 bit.

– If X is set, the fixed header is followed by exactly one header extension (defined by profile)

14

Header (cont)

• CSRC count (CC): This field is of length 4 bits.

– indicates the number of CSRC identifiers that follow the fixed headers. – the field has a non-zero value only if passed through a mixer.

• Marker bit (M): This field is of length 1 bit.

– If M is set, it indicates some significant events like frame boundaries to be marked in the packet stream. – For example, an RTP marker bit is set if the packet contains a few bits of the previous frame along with the current frame.

• Payload type (PT): This field is of length 7 bits.

– PT indicates the payload type carried by the RTP packet. – RTP Audio Video Profile (AVP) contains a default static mapping of payload type codes to payload formats. – Additional payload types can be registered with IANA.

15

Header (cont)

• Sequence number: This field is of length 16 bits.

– The number increments by one for each RTP data packet sent, with the initial value set to a random value. – The receiver can use the sequence number not only to detect packet loss but also to restore the packet sequence.

• Time stamp: This field is of length 32 bits.

– The time stamp reflects the sampling instant of the first octet in the RTP data packet. – The sampling instant must be derived from a clock that increments monotonically and linearly in time to allow synchronization and jitter calculations at the receiver. – The initial value should be random, so as to prevent known plain text attacks. – several packets may have equal timestamps (eg. same video frame), or even in disorder E.g. interpolated frames in MPEG

16

Header (cont) SSRC: This field is of length 32 bits. – This field identifies the source that is generating the RTP packets for this session. – The identifier is chosen randomly so that no two sources within the same RTP session have the same value. CSRC list: – The list identifies the contributing sources for the payload contained in this packet. – The maximum number of identifiers is limited to 15, as is apparent from the CC field

17

Synchronization Source (SSRC)

– Source of a stream of RTP packets – 32-bit numeric identifier, randomly chosen meant to be globally unique – Carried in RTP header (thus not dependent on network address) – Allows receiver to group packets from same synchronization source

To identify the use of the same timing & sequence number space

– A participant need not use the same SSRC identifier for all RTP sessions in a multimedia session

Binding of identifiers is provided through RTCP

– If a participant generates multiple streams in one RTP session (separate video cameras) each must be identified as a different SSRC

18

SSRC Identifier Allocation 32 bit random number

– Must not just pick IP address, or other local identifier – Must not just call random() without carefully initializing state

If done right, chance of collision is very small

– For joining a 1000-user session, about 1 in 5 million

However, RTP requires collision detection and resolution code

– If a source sees another with same SSRC, it sends a BYE and picks again – Receivers can temporarily supress a conflicting SSRC using transport address

Misconfigured translators/mixer can cause loops

19

Contributing Source (CSRC) A source of a stream of RTP packets that has contributed to the combined stream produced by an RTP mixer

20

RTP Use Scenarios Simple multicast audio conference

– A multicast group address is assigned – 2 UDP ports are assigned

One for RTP packaged audio data One for RTCP

– Address & port info distributed to intended participants – Encryption key is also distributed if privacy is required – Allocation and distribution mechanisms are outside the scope of RTP

21

RTP Use Scenarios (cont.)

– Audio conferencing application at each participant sends audio data in small chunks (e.g. of 20 msec duration)

RTP header is appended to each chunk

• Indicates type of encoding (e.g. PCM, ADPCM, LPC)

Senders can change encoding during a conference. E.g. – To accommodate a new participant connected thru a low bandwidth link – To react to network congestion

• Carries timing information (for playback purposes)

– Performed separately for each source

• Carries sequence number

– To estimate how many packets are lost RTP packet (header + data) encapsulated into a UDP packet

22

RTP Use Scenarios (cont.) RTCP

– Dynamic participant group

Useful to know:

• Who is participating
• How well they are receiving audio data

– Audio application periodically multicasts:

A reception report

• How well the current speaker is being received
• Used to control adaptive encoding

The name of its user

– RTCP BYE packet sent when participant leaves conference

23

RTP Use Scenarios (cont.) Audio & Video Conference

– Separate RTP sessions & RTCP packets for each medium (audio & video)

Different UDP ports pairs/multicast addresses Use same distinguished (canonical) name for both sessions in RTCP packets to associate the two sessions Separation allows participants to receive only one medium

– Synchronized playback of a source’s audio & video achieved using timing info carried in the RTCP packets for both sessions

24

RTP Use Scenarios (cont.) Mixers:

Used to mix multiple audio streams into one to handle a low bandwidth link – Resync incoming audio packets – Reconstruct 20 msec spacing generated by senders – Mix the reconstructed audio stream into a single stream – Translate audio encoding into a lower bandwidth encoding – Forward lower bandwidth packet streams on their way – RTP header includes:

Means to identify sources contributing to a mixed stream

• Correct talker indication can be provided at receiver

25

RTP Session Association among a set of participants communicating with RTP

– A session is defined by a particular pair of destination transport addresses

One network address A pair of ports (one RTP & one RTCP) May be common to all participants (as in IP multicast) May be different for each (individual network address + common port address)

26

Multiplexing RTP Sessions In RTP, multiplexing is provided by the destination transport address (combination of network address + port Multiplexing/ de-multiplexing based on payload type and/or SSRC field is not intended

– Payload type may be switched during a session – SSRC identifies a single timing and sequence number space

Different clock rates for different streams

27

Multiplexing RTP Sessions (cont.)

– RTCP sender & receiver reports describe only one timing & sequence space per SSRC – RTP mixer would not be able to combine interleaved streams of incompatible media in one session precludes:

Use of different network paths Use of different network resource allocation Reception of desired media (audio, not video)

28

RTP Translators & Mixers Intermediate systems acting as gateways at RTP layer

– Allow RTP traffic to pass through an Internet firewall – Mix and/or recode data to fit over a low bandwidth link

Translators forward RTP packets with original SSRC

– Typically used in firewall case – Allows sources to remain distinct even though packets all appear to come from the translator’s network address – May also change payload type or combine multiple packets into one

Mixers combine together streams from multiple sources

– Output of mixer has a newly assigned SSRC with new seq/timestamps – Original sources are listed in each packet in the CSRC list

29

Encryption support Application-level encryption of full packet recommended

– RTP / RTCP header contains padding bit to recover original packet length – Random start values of sequence number prevent known plaintext attacks – Decryption verified by sanity checking fields in header – Algorithm choice left to application by external means – Network layer encryption can be used where available

No explicit RTP support at this time for authentication

– Authentication could be accomplished via encapsulation

RTP can also be used in the clear carrying encrypted payloads

– Good for hardware which both decrypts & processes payload data

30

RTP Session

Mixer is an intermediate system that combines RTP streams from different sources into a single stream. It can change the data format of the RTP packets.

31

RTCP Protocol specifies report packets exchanged between sources and destinations of multimedia information Three reports are defined:

– Receiver report (RR), Sender report (SR), and Source description (SDES)

Reports contain statistics such as the number of packets sent, number of packets lost, inter-arrival jitter Used:

– to modify sender transmission rates – for diagnostics purposes

32

RTCP (cont.) Sent on separate port from data

– Monitoring tools can get info without sorting thru all data packets

Multiple RTCP segments can be combined in one packet

– Overall packet rate is reduced, saving both bandwidth & processing cost – No explicit count needed – overall length provided at lower layer

All packets must contain reception report information

– Losses in multicast distribution can be quickly isolated – Senders can adapt to current network conditions

All packets contain “CNAME” information

– Sources retain identity over time even if a collision occurs in SSRC – RTP streams from multiple tools can be bound together

33

RTCP Bandwidth Allocation Application decides how much bandwidth to use

– Might be included in a session announcement or inferred by scope

Control bandwidth should be fixed fraction of the total

– RTP currently recommends 5% of total session bandwidth – Deviations from this should be specified in the profile documents

RTCP bandwidth kept constant by varying report interval

– Each participant keeps track of number of active senders & receivers – Senders get 25% of total RTCP bandwidth, receivers get the rest – Minimum report interval is 5 secs., to avoid bursts on small sessions – Actual interval is randomized to avoid unintentional synchronization

34

RTCP

• Functions

– supervise the network QoS – flow control and congestion control – identification of participants – persistent id (CNAME = Canonical Name) – determine the number of participants – session information – traffic of RTCP < 5%

• Format of RTCP packets

– SR : sender reports

information on the source source statistics

– RR : reception reports

receiver statistics

– SDES : source description

CNAME

– BYE : end of the participation – APP : application specific functions

35

RTCP Sender Report (SR)

36

RTCP Reception Report (RR)

37

RTCP Source Description (SDES)

38

Other RTCP Segments

39

SR & RR Information on the source (only in SR)

– absolute timestamp (NTP) – timestamp (RTP) – number of packets sent RTP – number of bytes sent RTP

Statistics report for source SSRC-1 Statistics report for source SSRC-2 ... Statistics report for source SSRC-n

40

Statistics Report SSRC-n

– Fraction of lost packets – Number of lost packets – Last sequence number received – Estimation of the jitter – Timestamp of the last SR received (LSR) – Delay since the last SR received (DLSR)

41

Analyzing RTCP Reports Cumulative counts allow both long & short term analysis

– Any two reports can be differentiated to get activity over an interval – NTP timestamps in reports allow computation of rates – Monitoring tools need not know anything about particular encodings

Sender reports give utilization information

– Average packet rate & average data rate over any interval – Monitoring tools can compute this without reading any of the data

Receiver reports give loss & round-trip information

– Extended sequence number can be used to compute packets expected – Packets lost & packets expected give long term loss rate – Fraction lost filed gives short term loss rate, with only a single report – LSR & DLSR give sender’s ability to compute round trip time

42

Jitter Estimation Si - RTP timestamp RTP of packet i Ri - reception instant of packet i Di - jitter estimation for packet i

– Di = (Ri - Ri-1) – (Si - Si-1 )

Ji - temporal average of the jitter for packet i

– Ji = 15/16 Ji-1 + 1/16 | Di |

Used for adaptive playout

43

RTP Profiles Provides interpretation of generic fields Items which may be specified in a profile

– Default mapping between payload type numbers & encodings – Use of marker bits – Frequency of the RTP clock used to generate timestamps

44

RTP Profiles (cont.) Other items which maybe specified in a profile

– RTP header extension – Additional RTCP packet types – RTCP report interval – Use of security / encryption

No support for negotiation of parameters or membership control

45

Profile of Audio / Video Conferences with Minimal Control

Only that profile has been defined Specifies payload type codes for audio & video data Follows the default and/or recommended aspects of RTP specifications Uses the standard RTP header with one marker bit Static payload types No RTP header extensions or additions

46

Registering Payload Types Name & description of encoding RTP timestamp clock rate Who has control over the encoding Operating parameters or profiles Reference to a further description Payload type value if necessary

– Dynamic payload type assignment possible – Static payload type only if encoding is beneficial to entire Internet community

47

Audio Profile Marker bit is used to indicate the end of a silence (beginning of a talkspurt) RTP clock rate is equal to the sampling frequency

– 8000, 11025, 16000, 22000, 24000, 32000, 44100, 48000 Hz

Default packetization interval: 20 ms Defines payload codes or 15 audio codecs including:

– G.711, G.723 – GSM based – µ-law, A-law – …

48

Video Profiles Only specifies payload type codes Use of marker bits, clock rate & other parameters specified in payload definations Supports

– H.261, H,263 – MPEG 1 / MPEG 2 – JPEG – CelB – …

49

Payload Formats Defines how a particular payload, such as video, should be carried in RTP Specifies how data may be fragmented for encapsulation Defines a payload specific header which appears at the beginning of the RTP payload

50

H.261 Payload Format Defines a payload format which contains information which would be needed to recover from packet loss

– Such as the previous value of the motion vector

Timestamp indicates the time at which the first frame in the RTP packet was sampled RTP clock rate is 90 kHz (multiple of 29.97 fps) Marker bit set to 1 for the last packet of a video frame

51

H.261 Specific Header

– The H.261 information is carried as payload data within the RTP packet

52

H.261 Specific Header (cont.) The following fields of the RTP header are specified

– Payload type is set to H.261 payload format – RTP timestamp encodes the sampling instant of the first video image contained in the RTP packet

Will be the same for all RTP packets that belong to the same image Packets carrying data from different frames must have different timestamps Frames may be distinguished by timestamp If multiple frames are contained in one RTP packet, the timestamp refers to the first frame

• Other frames timing info can be extracted from the H.261 header

– The RTP header’s marker bit is set to 1 in the last packet of a video frame

Otherwise must be zero

53

H.261 Specific Header (cont.)

– Start Bit Position (SBIT, 3 bits):

Indicates the number of most significant bits that should be ignored in the first data octet

– End Bit Position (EBIT, 3 bits):

Indicates the number of least significant bits that should be ignored in the last data octet

– Intra Frame Encoded (I, 1 bit):

Set to 1 if this stream contains only INTRA-coded blocks Set to 0 if the stream may or may not contain INTRA-coded blocks This bit may not change during the course of an RTP session

54

H.261 Specific Header (cont.)

– Motion Vector Flag (V, 1 bit):

Set to 0 if the motion vectors are not used in this stream Set to 1 if motion vectors may or may not be used in this stream This bit may not change during the course of an RTP session

– GOB Number (GOBN, 4 bits):

Encodes the current GOB number at the start of the packet Set to 0 if packet begins with a GOB header

– Macroblock Address Predictor (MBAP, 5 bits):

Indicates the address of the last MB encoded in the previous packet Ranges from 0 to 32 to predict the valid MBAs 1-33 Since the bitstream cannot be fragmented between the GOB header and the first macroblock (MB 1), the predictor at the start of a packet can never be zero. Thus the range is 1-32 which is made to fit in 5 bits by biasing the value by -1

• i.e. if MBAP is 0, the value of the MBA predictor is 1

MBAP is set to 0 if the packet starts with a GOB header

55

H.261 Specific Header (cont.)

– Quantizer (QUANT, 5 bits):

Quantizer value (MQUANT or GQUANT) in effect prior to the start of this packet

• Set to 0 if the packet begins with a GOB header

– Horizontal Motion Vector Data (HMVD, 5 bits):

Reference horizontal motion vector data (MVD) Set to 0 if :

• V flag is 0 or if the packet starts with a GOB header
• When the MTYPE of the last MB encoded in the previous packet was not MC
• HMVD is encoded to 2’s complement form (value -16 (1000) is forbidden)

– Vertical Motion Vector Data (VMVD, 5 bits):

Reference vertical motion vector data (MVD) Set to 0 if :

• V flag is 0 or if the packet starts with a GOB header
• When the MTYPE of the last MB encoded in the previous packet was not MC
• VMVD is encoded to 2’s complement form (value -16 (1000) is forbidden)

I & V are “hint” flags as they can be inferred from the encoded bitstream. They are included to help the decoder make otherwise impossible optimizations based on these hints. These bits cannot change for the duration of the stream

56

H.261 Payload Packetization When encapsulating H.261 data into RTP packets two types of fragmentation is recommended:

– Fragmentation on GOB boundary – Fragmentation on MB boundary

57

H.261 Specific RTCP Messages In order to recover quickly from lost packets 2 new RTCP messages are introduced Used to request the sender encode portions of the next frame using intra-coding

58

H.261 Specific RTCP Messages (cont.) Full Intra-Frame Request (RTCP_FIR)

– Requests the sender encode the next full frame using intra-coding

Negative Acknowledgement (RTCP_NACK)

– Notifies the sender that certain RTP packets were lost – The encoder should encode the GOBs in the next frame which correspond to the lost GOBs using intra-coding

59

MPEG1 / MPAG2 Payload Format Defines two methods of transmitting MPEG1/2 video using RTP Encapsulate MPEG system and transport streams

– Provides maximum interoperability among MPEG systems

Encapsulate elementary streams (video & audio) separately

– Provides interoperability with other Internet-based end systems

60

Encapsulation of MPEG System & Transport Streams MPEG standard already contains System specification for encapsulating audio & video into a single stream Treat the MPEG systems streams as byte streams and encapsulate them without any specific RTP payload header or fragmentation restrictions

61

Encapsulation of MPEG System & Transport Streams (cont.)

Timestamp derived from a 90 kHz clock Synchronized to MPEG Program Clock Reference or System Clock Reference RTP timestamp only used to estimate & correct network-induced jitter – not sent to decoder M bit indicates discontinuities in timestamp

– E.g. sender switches data sources

62

Encapsulation of MPEG Elementary Streams Allows RTP to transport MPEG ½ Audio & Video streams separately Different RTP payload types assigned for audio & video No distinction between MPEG 1 & 2

63

MPEG Video-Specific Header

– MBZ: unused, reserved for future use. Must be zero – TR (10 bits):

Temporal reference of the current frame within the current GOP Value ranges between 0 & 1023 Same in all RTP packets belonging to the same frame

– Sequence Header Present (S, 1 bit):

Set to 1 when the RTP packet contains a sequence header

64

MPEG Video-Specific Header (cont.)

– Beginning of Slice (B, 1 bit – BS):

• Set to 1 when the start of the payload is a slice start code or when a slice

start code is preceded by only one or more Video _sequence_header, GOP_header and/or Picture_header

– End of Slice (E, 1 bit –ES)

• Set to 1 when the last byte of the payload is the end of an MPEG slice

– Picture Type (P, 3 bits):

• Type is I (1), P (2), B (3) or D (4) and is identical for all RTP packets of a

given picture

– FBV (1 bit):

• Full_pel_backward_vector

– BFC (3 bits):

• Backward_f_code

65

MPEG Video-Specific Header (cont.)

– FFV (1 bit):

• Full_pel_forward_vector

– FFC (3 bits):

• Forward_f_code

– FBV, BFC, FFV and FFC are obtained from the most recent picture header

• Constant for all packets belonging to the same picture
• I frame : all of these are set to zero in RTP header since they are not used
• P frame: FBV & BFC are set to zero in RTP header
• B frame: all four values are present

66

MPEG Video Stream Fragmentation Sequence Header must occur at the beginning of the RTP payload GOP header must occur at beginning of RTP payload

• r immediately following a Sequence Header

Picture header must occur at beginning of RTP pyload

• r after the GOP Header

Slice must occur at the beginning of the RTP packet

67

MPEG Audio-Specific Header (cont.)

– MBZ: unused, reserved for future use. Must be zero – Fragment Offset (16 bits):

• Indicates the offset (in bytes) of the data in this packet into the audio frame

68

RTSP (Real-Time Streaming Protocol) Similar to HTTP

– rtsp://cs.lums.edu.pk/class_video

Description of available media

– SDP (Session Description Protocol)

Allows to establish RTP sessions Session control

– start, pause, resume, end