Network-Integrated Multimedia Middleware KDE aKademy 2004 Marco - - PowerPoint PPT Presentation

▶
network integrated multimedia middleware
SMART_READER_LITE
LIVE PREVIEW

Network-Integrated Multimedia Middleware KDE aKademy 2004 Marco - - PowerPoint PPT Presentation

Oct 11, 2023 336 likes •894 views

Network-Integrated Multimedia Middleware KDE aKademy 2004 Marco Lohse, Michael Repplinger, Philipp Slusallek Computer Graphics Lab, Saarland University, Germany KDE aKademy 2004 - NMM p.1/46 Motivation camera display PDA Multimedia VCR

slide-1
SLIDE 1

Network-Integrated Multimedia Middleware

KDE aKademy 2004

Marco Lohse, Michael Repplinger, Philipp Slusallek Computer Graphics Lab, Saarland University, Germany

KDE aKademy 2004 - NMM – p.1/46

slide-2
SLIDE 2

Motivation

Internet

Multimedia PC

camera display

Mobile phone PC Set-top box Hi-fi equipment VCR PDA

  • PC-centric approach (back in 2000, today?)
  • Stand-alone multimedia-PC
  • Streaming predefined content from the Internet
  • Many other multimedia devices

⇒ Unexploited networking capabilities

KDE aKademy 2004 - NMM – p.2/46

slide-3
SLIDE 3

Motivation

........................................................................ Internet

Multimedia PC

camera display

Mobile phone PC Set-top box Hi-fi equipment VCR PDA

⇒ Network-Integrated Multimedia Middleware (NMM)

  • GNU/Linux (x86/ARM), LGPL/GPL, C++
  • http://www.networkmultimedia.org

KDE aKademy 2004 - NMM – p.3/46

slide-4
SLIDE 4

NMM

  • Network-Integrated
  • Extend cooperation and control to network
  • Multimedia (for GNU/Linux)
  • Uniform access to available drivers and libraries
  • Middleware
  • Layer between applications and distributed systems

Operating system Application NMM Operating system Network

KDE aKademy 2004 - NMM – p.4/46

slide-5
SLIDE 5

Application Scenarios

  • Access distributed resources
  • Distributed transcoding
  • Watch TV using remote TV receiver
  • Control remote TV receiver

Stationary system

Distributed processing power Distributed I/O capabilities

Stationary system

Processing power I/O capabilities TV receiver

KDE aKademy 2004 - NMM – p.5/46

slide-6
SLIDE 6

Collaborative Scenarios

  • Number of users simultaneously enjoys the same (or

similar) content using different devices

  • “Drive-in cinema”
  • Watch video of DVD on large screen
  • Synchronized playback on different PDAs

Stationary system

DVD playback

Mobile system 1

audio track 0

Mobile system 2

audio track 1 video

KDE aKademy 2004 - NMM – p.6/46

slide-7
SLIDE 7

Dynamic Scenarios

  • User and device mobility
  • Use capabilities of surrounding environment
  • Handover of media playback
  • E.g. use high-fidelity audio output

Stationary system Mobile system

MP3 playback

KDE aKademy 2004 - NMM – p.7/46

slide-8
SLIDE 8

Dynamic Scenarios

  • User and device mobility
  • Use capabilities of surrounding environment
  • Handover of media playback
  • E.g. use high-fidelity audio output

Stationary system Mobile system

MP3 playback

KDE aKademy 2004 - NMM – p.7/46

slide-9
SLIDE 9

Beyond Client/Server

  • Traditional client/server streaming ⇒ Black-box
  • Locate server? Activate server? Provided

functionality? Connection setup? Streamed data format? Control server, e.g. switch the TV channel? Complex application scenarios, e.g. transcode content? Distribute workload? Synchronization? Shared applications?

KDE aKademy 2004 - NMM – p.8/46

slide-10
SLIDE 10

Beyond Client/Server

  • Traditional client/server streaming ⇒ Black-box
  • Locate server? Activate server? Provided

functionality? Connection setup? Streamed data format? Control server, e.g. switch the TV channel? Complex application scenarios, e.g. transcode content? Distribute workload? Synchronization? Shared applications?

⇒ It’s not a question of“if”– it’s a question of“how” ⇒ Generally solve challenges within middleware ⇒ Provide suitable abstractions and programming model ⇒ Easy, efficient development of distributed applications

KDE aKademy 2004 - NMM – p.8/46

slide-11
SLIDE 11

Overview

  • Motivation
  • Architecture of NMM
  • Developing plug-ins for NMM
  • Basic middleware services
  • Registry service, Synchronization, GraphBuilder
  • Distributed multimedia application
  • helloworld, clic, Multimedia-Box, ...
  • Advanced Middleware Services
  • Application sharing, seamless handover
  • Summary, future work & discussion

KDE aKademy 2004 - NMM – p.9/46

slide-12
SLIDE 12

Flow Graph

  • Nodes as smallest processing unit
  • Jacks to connect nodes
  • Formats to type connections
  • Interfaces to control objects
  • Messages to forward media data or control information

Generic Read Node

Output Jack audio/mpeg audio/raw

Application

Output Jack Input Jack Input Jack

MPEG Audio Decode Node Playback Node

KDE aKademy 2004 - NMM – p.10/46

slide-13
SLIDE 13

Flow Graph

  • Nodes as smallest processing unit
  • Jacks to connect nodes
  • Formats to type connections
  • Interfaces to control objects
  • Messages to forward media data or control information

Generic Read Node

Output Jack audio/mpeg audio/raw INode

Application

Output Jack Input Jack Input Jack

MPEG Audio Decode Node Playback Node

INode INode

KDE aKademy 2004 - NMM – p.10/46

slide-14
SLIDE 14

Flow Graph

  • Nodes as smallest processing unit
  • Jacks to connect nodes
  • Formats to type connections
  • Interfaces to control objects
  • Messages to forward media data or control information

Generic Read Node

Output Jack audio/mpeg audio/raw INode

Application

Output Jack Input Jack Input Jack

MPEG Audio Decode Node Playback Node

INode INode ISeekable

KDE aKademy 2004 - NMM – p.10/46

slide-15
SLIDE 15

Flow Graph

  • Nodes as smallest processing unit
  • Jacks to connect nodes
  • Formats to type connections
  • Interfaces to control objects
  • Messages to forward media data or control information

INode

Application MPEG Audio Decode Node Playback Node

INode INode B B C E E E

Generic Read Node

C B B

  • ut-of-band

downstream upstream

KDE aKademy 2004 - NMM – p.10/46

slide-16
SLIDE 16

Messaging System

  • Message
  • Header information such as timestamps
  • Buffer
  • Multimedia data
  • Efficiently managed by buffer pools
  • Composite event (CEvent)
  • Stores list of events
  • Event
  • Key, list of values, return value, exception
  • Represents single method call for dispatcher
  • Both, out-of-band and instream interaction

KDE aKademy 2004 - NMM – p.11/46

slide-17
SLIDE 17

Flow Graph

  • NMM flow graphs are distributed
  • Local and remote nodes can be controlled and

integrated into a common flow graph

  • Transparent for developers, no overhead for all

co-located parts of flow graph

Generic Read Node

Output Jack INode

Application

Output Jack Input Jack Input Jack

MPEG Audio Decode Node Playback Node

INode INode

KDE aKademy 2004 - NMM – p.12/46

slide-18
SLIDE 18

Flow Graph

  • NMM flow graphs are distributed
  • Local and remote nodes can be controlled and

integrated into a common flow graph

  • Transparent for developers, no overhead for all

co-located parts of flow graph

Generic Read Node

Output Jack INode

Application

Output Jack Input Jack Input Jack

MPEG Audio Decode Node Playback Node

INode INode

KDE aKademy 2004 - NMM – p.12/46

slide-19
SLIDE 19

Interfaces

  • NMM Interface Definition Language (NMM IDL)
  • Similar to CORBA IDL
  • Interface class and implementation skeleton
  • Supports inheritance, exceptions, arbitrary data

types, state machine, ...

  • Both, out-of-band and instream interaction

KDE aKademy 2004 - NMM – p.13/46

slide-20
SLIDE 20

Interfaces

module NMM { interface IFileHandler { void setFilename(in string name); Result endTrack() instream; }; }

IFileHandler

+setFilename(const string&): void +endTrack(): Result +create_endTrack(): Event*

Code generation IFileHandler.idl IFileHandlerImpl

+setFilename(const string&): void +endTrack(): Result

ConcretePluginNode

+setFilename(const string&): void +endTrack(): Result

KDE aKademy 2004 - NMM – p.14/46

slide-21
SLIDE 21

Nodes

  • Different types of nodes & generic base classes

Demux Node Mux Demux Node Mux Node

Filter / Converter Node

Source Node Sink Node

  • State machine controls life-cycle

CONSTRUCTED INITIALIZED OUTPUT INITIALIZED ACTIVATED STARTED init(); doInit(); initOutput(); doInitOutput(); activate(); doActivate(); start(); doStart(); deinit(); doDeinit(); deinitOutput(); doDeinitOutput(); deactivate(); doDectivate(); stop(); doStop(); Constructor();

KDE aKademy 2004 - NMM – p.15/46

slide-22
SLIDE 22

60+ Plug-ins

  • Source
  • CDDA, DVD/menus, WinTV PVR, KFIR, DVB, Firewire,

VISCA cameras, audio devices, WAV, AVI

  • Encoder/decoder/converter/filter
  • MPEG audio, MPEG video, DivX, Ogg/Vorbis, SPU, PNG,

JPEG, RTJPEG

  • Color space, video scaler, on-screen menus, deinterlacer
  • Multiplexer/demultiplexer
  • AVI, MPEG, OGM
  • Sink
  • Video output via X or OpenGL, audio devices, WAV, AVI

KDE aKademy 2004 - NMM – p.16/46

slide-23
SLIDE 23

Communication Channels

Node Application Interface

Output Jack B C B Buffer B Composite event C Event E B B C E E Communication Channel Communication Channel Input Jack

Network

KDE aKademy 2004 - NMM – p.17/46

slide-24
SLIDE 24

Communication Channels

  • Serialization and transport of objects
  • E.g. Buffers, composite events, events
  • Scalable transparency
  • Automatic setup vs. manual configuration
  • Serialization strategies
  • Magic number, XML, ...
  • Transport strategies
  • Pointer forwarding, TCP, UDP, RTP, ...
  • Local optimization
  • Pointer forwarding or elimination of communication

channel

KDE aKademy 2004 - NMM – p.18/46

slide-25
SLIDE 25

Developing Plug-ins

  • Specification
  • Supported functionality? Features?
  • Granularity and base classes
  • How many nodes? What kind of node?
  • Implementation of (some) state transitions
  • doInit() : generally supported formats
  • doInitOutput() : currently supported formats
  • Definition and implementation of interfaces
  • Out-of-band and instream interaction
  • Implementation of processBuffer()
  • Register node with registry service

KDE aKademy 2004 - NMM – p.19/46

slide-26
SLIDE 26

processBuffer()

Message∗ MyConverterNode::processBuffer(Buffer ∗ in buffer ) { // get new buffer with defined size Buffer∗ out buffer = getNewBuffer( out buffer size ) ; // get data of buffer char∗ p = in buffer−>getData(); // some code ... // release in buffer since it is no longer needed in buffer −>release(); return out buffer ; }

KDE aKademy 2004 - NMM – p.20/46

slide-27
SLIDE 27

processBuffer()

Message∗ MySinkNode::processBuffer(Buffer ∗ in buffer ) { // release in buffer since it is no longer needed in buffer −>release(); // create composite event including single event CEvent∗ cevent = new CEvent(IExample::create foo()); // set direction to UPSTREAM cevent−>setDirection(Message::UPSTREAM); return cevent; }

KDE aKademy 2004 - NMM – p.21/46

slide-28
SLIDE 28

Event Handling

  • Automatic registration and dispatching due to

implementation of interface

  • Out-of-band and instream interaction

// Implementation of IFileHandlerImpl :: endTrack() Result MySinkNode::endTrack() { // handle endTrack, e.g . close file return SUCCESS; }

KDE aKademy 2004 - NMM – p.22/46

slide-29
SLIDE 29

Developing Plug-ins

  • Modify, add, delete, replace events
  • Handling multiple input and output jacks
  • Handling arbitrary patterns of incoming and outgoing

messages

  • Upstream and downstream traffic
  • working-flag and producing-flag
  • ...

⇒ See“Developing Plug-ins for NMM”

KDE aKademy 2004 - NMM – p.23/46

slide-30
SLIDE 30

helloworld

// MP3 player: the NMM application for the example NMMApplication∗ app = ProxyApplication :: getApplication (argc , argv); // create the node descriptions NodeDescription readfile (”GenericReadNode” ); NodeDescription decoder(”MPEGAudioDecodeNode” ); NodeDescription audioplay(”PlaybackNode” ); // create a query for a flow graph GraphDescription graph; graph.addEdges(&readfile, &decoder, &audioplay);

KDE aKademy 2004 - NMM – p.24/46

slide-31
SLIDE 31

helloworld

// request complete graph from registry ClientRegistry & registry = app−>getRegistry(); registry .requestGraph(graph); // set the filename by requesting appropriate interface INode∗ inode = graph.getINode( readfile ); IFileHandler var filehandler (inode−>getParentObject()

−>getCheckedInterface<IFileHandler>());

filehandler −>setFilename(argv[1]);

KDE aKademy 2004 - NMM – p.25/46

slide-32
SLIDE 32

helloworld

// realize and start graph

  • graph. realizeGraph ();

graph.startGraph(); // wait ... // stop complete graph graph.stopGraph(); // release complete graph registry .releaseGraph(graph);

KDE aKademy 2004 - NMM – p.26/46

slide-33
SLIDE 33

helloNMM

// Distributed MP3 player: // decoder and audio sink on remote host argv[2] NodeDescription readfile (”GenericReadNode” ); NodeDescription decoder(”MPEGAudioDecodeNode” ); NodeDescription audioplay(”PlaybackNode” ); // NEW : set location for decoder and audio sink decoder.setLocation(argv [2]) ; audioplay . setLocation(argv [2]) ; GraphDescription graph; graph.addEdges(&readfile, &decoder, &audioplay);

KDE aKademy 2004 - NMM – p.27/46

slide-34
SLIDE 34

helloNMM

ClientRegistry & registry = app−>getRegistry(); // NEW : add remote host to registry // ( ./ serverregistry is running on remote host ) registry .addRegistry(TCPAddress(argv[2], ClientRegistry :: default port ) , argv [2]) ; // same code as before ... registry .requestGraph(graph);

  • graph. realizeGraph ();

graph.startGraph();

KDE aKademy 2004 - NMM – p.28/46

slide-35
SLIDE 35

Further Options

  • Node descriptions
  • Types, I/O formats, interfaces, sharing policy
  • Manual creation of flow graphs
  • connect(node1, node2);
  • Configuration of communication channels
  • c connect(node1, node2);
  • Listener notification
  • ...

⇒ See“Hello World! Welcome to NMM Application Devel-

  • pment :)”

KDE aKademy 2004 - NMM – p.29/46

slide-36
SLIDE 36

Registry Service

  • Administrates locally available nodes
  • Queries
  • Single (subset) of a node description, or
  • Graph description
  • Peer-to-peer approach for distributed flow graphs

Generic Read Node

MPEG Audio Decode Node

Playback Node

Host 2 Host 3 Host 3 SOURCE audio/raw IFileSource Type Interfaces Name Input format Output format Location CONVERTER SINK audio/raw audio/mpeg

  • KDE aKademy 2004 - NMM – p.30/46
slide-37
SLIDE 37

Registry Service

  • Administrates locally available nodes
  • Queries
  • Single (subset) of a node description, or
  • Graph description
  • Peer-to-peer approach for distributed flow graphs

Registry

Network

Registry

Host 2

Registry

Host 3 Host 1

Generic Read Node

MPEG Audio Decode Node

Playback Node

Host 2 Host 3 Host 3 SOURCE audio/raw IFileSource Type Interfaces Name Input format Location CONVERTER SINK audio/raw audio/mpeg

  • KDE aKademy 2004 - NMM – p.30/46
slide-38
SLIDE 38

clic

  • Command line interaction and configuration
  • Setting up distributed flow graphs from textual

descriptions

  • ./clic <.gd file> [-i <input file>] [-o <output file>]
  • Examples

mp3play.gd:

GenericReadNode ! MPEGAudioDecodeNode ! PlaybackNode

./clic mp3play.gd -i /home/bob/music/song.mp3

KDE aKademy 2004 - NMM – p.31/46

slide-39
SLIDE 39

dvbtv.gd

% Watch TV via DVB receiver in remote host DVBReadNode #Event ” setChannel(1)”ACTIVATED #Host ” host.domain.org” ! MPEGDemuxNode { { [ ”mpeg audio0”] ! MPEGAudioDecodeNode ! PlaybackNode } { [ ”mpeg video0”] ! MPEGVideoDecodeNode ! XDisplayNode } }

KDE aKademy 2004 - NMM – p.32/46

slide-40
SLIDE 40

GraphBuilder

  • Automatically creates distributed flow graph for media

playback

  • Distributed source, audio sink, and video sink
  • Supported URLs
  • file, audiocd, dvd, tv, dvbtv, ivtv, mpegtv
  • Examples

./clic -u file:///home/bob/mp3/song.mp3 ./clic -u ” dvd:///dev/cdrom?title=1&chapter=4&angle=1” ./clic -u file://host1/movies/movie.mpeg -A host2 -V host3

⇒ Demo

KDE aKademy 2004 - NMM – p.33/46

slide-41
SLIDE 41

Distributed Synchronization

  • Strict separation
  • Locally running controllers
  • Synchronizer globally adjusts latencies
  • Global clock via NTP

Synchronizer SyncSinkNode

SinkController

SyncSinkNode

SinkController

Network/address space

InputJack

T

B

T

B B

T

InputJack

T

B

T

B B

T

B

Node

T T

B B

Node

T T

B

KDE aKademy 2004 - NMM – p.34/46

slide-42
SLIDE 42

Applications

  • Virtual Courseroom Environment (VCORE)
  • Record and broadcast all aspects of a talk
  • Audio and video stream of the speaker, slides,

written text or annotations (SMIL)

  • Successfully used at Saarland University since

winter 2002

KDE aKademy 2004 - NMM – p.35/46

slide-43
SLIDE 43

Applications

  • amaroK – the audio player for KDE
  • NMM engine employs GraphBuilder (next NMM

release)

⇒ Demo

KDE aKademy 2004 - NMM – p.36/46

slide-44
SLIDE 44

Multimedia-Box

  • Goal (in April 2001)
  • Networked multimedia home entertainment
  • Developed completely on top of NMM
  • E.g. plug-in for on-screen display
  • Seamless integration of mobile devices

KDE aKademy 2004 - NMM – p.37/46

slide-45
SLIDE 45

Multimedia-Box

  • CD-Player with CDDB support
  • CD-Grabber and transcoding to MP3/OggVorbis
  • DVD-Player with support for menus
  • DVD-Grabber and distributed transcoding
  • TV, time-shifting, access to remote receiver
  • Video recorder and Electronic Program Guide (EPG)
  • Media player with playlist
  • Multi-tasking: watch TV while transcoding a DVD
  • Controllable with a remote-control
  • Extensible and configurable application framework (XML)

KDE aKademy 2004 - NMM – p.38/46

slide-46
SLIDE 46

Multimedia-Box

⇒ Demo

KDE aKademy 2004 - NMM – p.39/46

slide-47
SLIDE 47

Session Sharing

  • Service that allows to“overlap”running flow graph with

query

  • E.g. shared access to DVD, different audio tracks

for mobile devices

MPEG Demux Node Audio Decode Node Playback Node Video Decode Node Display Node DVDRead Node MPEG Demux Node Audio Decode Node Playback Node DVDRead Node

Running session 1

localhost localhost

Query

"audio1" "audio0" "video"

Synchronizer

KDE aKademy 2004 - NMM – p.40/46

slide-48
SLIDE 48

Session Sharing

  • Service that allows to“overlap”running flow graph with

query

  • E.g. shared access to DVD, different audio tracks

for mobile devices

MPEG Demux Node DVDRead Node MPEG Demux Node Audio Decode Node Playback Node Video Decode Node Display Node DVDRead Node Audio Decode Node Playback Node

Running session 1

localhost localhost

Running session 2

"audio1" "audio0" "video"

Synchronizer

KDE aKademy 2004 - NMM – p.40/46

slide-49
SLIDE 49

Seamless Handover

  • Playback session running on mobile system
  • Handed over to nearby stationary system
  • Seamless and synchronized handover
  • No loss, no duplicates, no interruption

Readfile Node Audio Decode Node Playback Node Audio Decode Node Playback Node

Mobile system Stationary system

KDE aKademy 2004 - NMM – p.41/46

slide-50
SLIDE 50

Summary

  • Network-Integrated Multimedia Middleware (NMM)
  • Research project and Open Source project
  • Simple, clear, and unified design
  • Nodes connected by jacks that stream messages
  • Unified out-of-band and instream interaction
  • Object-oriented design
  • Inheritance for interfaces
  • Integrating micro-core architecture
  • Middleware core plus plug-ins, services, transport

and serialization strategies

  • No overhead for all co-located components

KDE aKademy 2004 - NMM – p.42/46

slide-51
SLIDE 51

Summary

  • Scalable transparency
  • Network as black-box vs. tuning of network

parameters

  • Unique features
  • Transparently distributed flow graphs
  • Distributed GraphBuilder
  • Distributed synchronization
  • Session sharing service
  • Seamless handover
  • Research project with real world applications
  • Multimedia-Box, VCORE

KDE aKademy 2004 - NMM – p.43/46

slide-52
SLIDE 52

Future Work

  • Update plug-ins
  • deb/rpm packages
  • Optimization of current implementation
  • Support for standard protocols like SIP
  • Distributed streaming server with overlay network
  • Intelligent distribution of workload, e.g. for transcoding
  • (Help with) support for other platforms
  • (Help with) integration into KDE (and other projects)

KDE aKademy 2004 - NMM – p.44/46

slide-53
SLIDE 53

Acknowledgments

NMM contributors: Andreas Meyer, Benjamin Deutsch, Christian Ger- stner, Christoph Wellner, David Maass, David Philippi, Eric Peters, Florian Winter, Marc Klein, Markus Sand, Patrick Becker, Patrick Cernko, Patrick Wambach, Robert Wruck, Roger Dostert, Stephan Di- das, Wolfgang Enderlein, and Wolfram von Funck; Georg Demme (who supervises the VCORE project and designed the NMM logo), Andreas Pomi (helps managing the technical infrastructure at our lab).

KDE aKademy 2004 - NMM – p.45/46

slide-54
SLIDE 54

Questions?

KDE aKademy 2004 - NMM – p.46/46