Intel SMD elements in GStreamer Josep Torra a.k.a. ad-n770 - - PowerPoint PPT Presentation

▶
intel smd elements in gstreamer
SMART_READER_LITE
LIVE PREVIEW

Intel SMD elements in GStreamer Josep Torra a.k.a. ad-n770 - - PowerPoint PPT Presentation

Oct 04, 2023 420 likes •769 views

Intel SMD elements in GStreamer Josep Torra a.k.a. ad-n770 josep@fluendo.com Sodaville and Canmore Intel CE4100 Media Processor, Sodaville Intel CE3100 Media Processor, Canmore SoC with HW accelerated codecs Targeted to Consumer Electronics

slide-1
SLIDE 1

Intel SMD elements in GStreamer

Josep Torra a.k.a. ad-n770 josep@fluendo.com

slide-2
SLIDE 2

Sodaville and Canmore

Intel CE4100 Media Processor, Sodaville Intel CE3100 Media Processor, Canmore SoC with HW accelerated codecs Targeted to Consumer Electronics devices Intel CESDK, a minimalistic Linux system

slide-3
SLIDE 3

Media capabilities

Two High Definition video decoders capable

  • f 1080p@60

HW accelerated audio decoders/encoders

  • n DSPs and 7.1 audio support

Noise reduction, motion adaptive de- interlacing and scaling in HW Post-Processor HW blender for display composition OpenGL ES 1.1/2.0 capable GPU

slide-4
SLIDE 4

Intel Streaming Media Driver (ISMD)

Low-level driver which provides an API for application to control media streams. Devices, ports, memory ISMD buffers, ISMD events. A minimalistic media framework.

slide-5
SLIDE 5

Application Block Diagram

Hardware layer Fluendo SW Codecs GDL ISMD Fluendo ISMD elements GStreamer Media Application XPSM Media Application

slide-6
SLIDE 6

Fluendo ISMD elements (I)

Wraps ISMD devices Configurable through GObject properties GstPads connected to ISMD ports Converts GstBuffers into ISMD buffers and vice versa Wraps the GstStates to ISMD states

slide-7
SLIDE 7

Fluendo SMD elements (II)

Integrated with autoplug elements uridecodebin, decodebin2 and playbin2 Can be mixed with software elements like sources, demuxers or video decoders Transparent integration with ISMD BufMon Advanced features such as trick modes and handling of live pipelines

slide-8
SLIDE 8

DVB Source (I)

Wraps a ISMD Demuxer wired to TSI MPTS to SPTS filtering Performs clock recoveries based on PCR samples Capable to parse and manage PSI pids Integrated with ISMD BufMon

slide-9
SLIDE 9

DVB Source (II)

ismd_dvb_src

ISMD Buffer

ISMD Demuxer Device

TSI

GstBuffer

Main Memory Main Memory Clock Recovery Info ISMD Buffer

PSI Filter

PSI PIDs PATInfo GstMessage PMTInfo GstMessage ES PIDs

slide-10
SLIDE 10

Clock Recovery and Provider (I)

Provides a ISMD clock Integrated with ISMD BufMon Capable to parse the PCR samples and perform clock recoveries

slide-11
SLIDE 11

Clock Recovery and Provider (II)

ismd_clock_recovery_provider GstBuffer

Main Memory

GstClock GstBuffer

Main Memory

MPEG TS PCR parser

Clock Recovery Info

slide-12
SLIDE 12

Audio Sink (I)

Wraps ISMD Audio processors Provides audio rendering for raw PCM and compressed formats (MPEG audio, AAC, AC3, DTS, WMA) Permits passthrough and encode to AC3

  • r DTS on digital outputs
slide-13
SLIDE 13

Audio Sink (II)

HDMI, SPDIF, I2S0 and I2S1outputs Integrates with ALSA emulation deamon Provides an ISMD clock

slide-14
SLIDE 14

Audio Sink (III)

ismd_audio_sink

ISMD Audio Processor Device

GstBuffer

Main Memory ISMD Buffer HDMI SPDIF I2S0 I2S1

GstClock

slide-15
SLIDE 15

Video Decoder (I)

Wraps ISMD Video decoder Performs video decoding Supported formats are MPEG2, MPEG4 part 2 (only on Sodaville), H264 and VC1 Frame reordering and PTS interpolation

slide-16
SLIDE 16

Video Decoder (II)

ismd_h264_viddec GstBuffer

ISMD Buffer

ISMD Video Decoder Device

GstBuffer

Main Memory ISMD Buffer ISMD Buffer

slide-17
SLIDE 17

Video Post-Processor (I)

Wraps ISMD Video Post-Processor Performs chroma up-sampling, noise reduction filtering, de-interlacing, scaling and Pan&Scan Converts video frames in standard GstBuffers into ISMD buffers and provides pad alloc capability.

slide-18
SLIDE 18

Video Post-Processor (II)

ismd_vidpproc GstBuffer

ISMD Buffer

ISMD Video Post-Processor Device

ISMD Buffer ISMD Buffer

GstBuffer

ISMD Buffer

slide-19
SLIDE 19

Video Post-Processor (III)

ismd_vidpproc GstBuffer

ISMD Buffer

ISMD Video Post-Processor Device

ISMD Buffer ISMD Buffer

GstBuffer

Main Memory

NV12 convert

slide-20
SLIDE 20

Video Sink (I)

Wraps ISMD Video renderer Presents the video frames on the selected GDL plane Configurable destination rectangle Provides an ISMD clock

slide-21
SLIDE 21

Video Sink (II)

ismd_vidrend_sink

ISMD Video Render Device

ISMD Buffer ISMD Buffer

GstBuffer

ISMD Buffer QoS GstEvent

GstClock

slide-22
SLIDE 22

Video Bin

Helper GstBin for integration with playbin2 ismd_vidpproc ! ismd_vidrend_sink

slide-23
SLIDE 23

GDL Video Sink (I)

Derived from GstVideoSink Presents the ARGB video frames on the selected GDL plane Designed to overlay subtitles and subpictures

slide-24
SLIDE 24

GDL Video Sink (II)

flugdlsink

GDL Plane

GDL Surface

GstBuffer

Main Memory

slide-25
SLIDE 25

Source Code

Fluendo ISMD elements:

http://core.fluendo.com/gstreamer/src/

FLUB build system for CESDK

https://core.fluendo.com/gstreamer/trac/browser/trunk/flub-ismd/

Building guide

https://core.fluendo.com/gstreamer/trac/wiki/HowToBuildFromSvn

slide-26
SLIDE 26

Application Use Cases

slide-27
SLIDE 27

File playback pipeline

filesrc demuxer queue queue queue ismd_h264_viddec textrender flugdlsink queue ismd_audio_sink ismd_vidrend_bin ismd_vidpproc ismd_vidrend_sink

slide-28
SLIDE 28

Digital TV playback pipeline

flutsdemux queue queue ismd_mpeg2_viddec ismd_audio_sink ismd_vidrend_bin ismd_vidpproc ismd_vidrend_sink ismd_dvb_src

slide-29
SLIDE 29

IPTV playback pipeline

flutsdemux queue queue ismd_mpeg2_viddec ismd_audio_sink ismd_vidrend_bin ismd_vidpproc ismd_vidrend_sink udpsrc

ismd_clock_recovery_provider

slide-30
SLIDE 30

Fluendo Media Center aka Moovida Fluendo DVD playback engine Video playback on 3D surfaces

Conclusions and Future

slide-31
SLIDE 31

Q&A

slide-32
SLIDE 32

About Fluendo

The Fluendo group was created at the end of 2003 with the very ambitious goal of improving the multimedia experience in the Free Software environment. Indeed at that time there was no reliable solution for multimedia application developers. Existing media players were not legal and there was no media framework to compete with the existing operating systems as an alternative to Microsoft Windows with DirectShow and Mac OS X with Quicktime. The Fluendo choice was a Free Software project called GStreamer which aimed at being as good as DirectShow or even better and funded a deep refactoring of the framework to bring it up at a professional level. Based on this multimedia expertise Fluendo started to develop products around GStreamer such as: Flumotion Streaming Server, Fluendo codec pool, Fluendo DVD player and Fluendo Media Center (aka Moovida). Nowadays Fluendo provides a wide range of products and solutions on top of the GStremaer framework and offers its expertise in the form of consultancy services to the main industry players.

slide-33
SLIDE 33

The garden (1977)

Joan Miró i Ferrà (April 20, 1893 – December 25, 1983); was a Catalan painter, sculptor, and ceramicist born in Barcelona.