VLC audio/video outputs Linux.conf.au Multimedia & Music R emi - - PowerPoint PPT Presentation

vlc audio video outputs
SMART_READER_LITE
LIVE PREVIEW

VLC audio/video outputs Linux.conf.au Multimedia & Music R emi - - PowerPoint PPT Presentation

VLC audio/video outputs Linux.conf.au Multimedia & Music R emi Denis-Courmont VideoLAN project Auckland, 12th January 2015 Outline Pipeline Audio output Video output Forewords Pipeline Audio output Video output End Note well


slide-1
SLIDE 1

VLC audio/video outputs

Linux.conf.au Multimedia & Music R´ emi Denis-Courmont

VideoLAN project

Auckland, 12th January 2015

slide-2
SLIDE 2

Outline

Pipeline Audio output Video output

slide-3
SLIDE 3

Forewords Pipeline Audio output Video output End

Note well The opinions hereby expressed represent the personal views of the author.

slide-4
SLIDE 4

Forewords Pipeline Audio output Video output End

Attendees advisory

  • I speak fast.
  • I may not articulate adequately.
slide-5
SLIDE 5

Forewords Pipeline Audio output Video output End

Attendees advisory

  • I speak fast.
  • I may not articulate adequately.

Do interrupt me if needed.

slide-6
SLIDE 6

Forewords Pipeline Audio output Video output End

Outline

Pipeline Audio output Video output

slide-7
SLIDE 7

Forewords Pipeline Audio output Video output End

Pipeline

  • defacto standard among multimedia frameworks
  • enforced by the specifications
slide-8
SLIDE 8

Forewords Pipeline Audio output Video output End

Pipeline overview

  • byte stream reader
  • format parser
  • (packetizers)
  • decoders: audio, video, text
slide-9
SLIDE 9

Forewords Pipeline Audio output Video output End

Pipeline overview

  • byte stream reader
  • format parser
  • (packetizers)
  • decoders: audio, video, text
  • filters
  • blending / overlay
  • outputs
slide-10
SLIDE 10

Forewords Pipeline Audio output Video output End

Driving the pipeline

  • buffers here and there
  • rate control, drift compensation and lip sync
slide-11
SLIDE 11

Forewords Pipeline Audio output Video output End

Outline

Pipeline Audio output Video output

slide-12
SLIDE 12

Forewords Pipeline Audio output Video output End

Buffers

Media playback has audibly long buffers

  • avoids underruns due to scheduling:

no stutter

slide-13
SLIDE 13

Forewords Pipeline Audio output Video output End

Buffers

Media playback has audibly long buffers

  • avoids underruns due to scheduling:

no stutter

  • reduces (or even eliminates) periodic interrupts:

lower power

slide-14
SLIDE 14

Forewords Pipeline Audio output Video output End

Buffers

Media playback has audibly long buffers

  • avoids underruns due to scheduling:

no stutter

  • reduces (or even eliminates) periodic interrupts:

lower power

  • unlike games and UIs:

needs special support to avoid latency and glitches

slide-15
SLIDE 15

Forewords Pipeline Audio output Video output End

Buffer requirements

  • Play-out latency estimate
  • Maintain lip synchronization
  • Control upstream pipeline rate
  • Drain or fill levels estimate:
  • Normal EOS without loosing last samples
slide-16
SLIDE 16

Forewords Pipeline Audio output Video output End

Interactive requirements

  • Flush: user stop or exit
  • Pause/resume

(some APIs cycle their playout buffer)

  • Volume and mute control
  • interactive volume control
  • per stream (not whole device!)
slide-17
SLIDE 17

Forewords Pipeline Audio output Video output End

  • Misc. requirements
  • Device enumeration, hotplug events
  • Configuration and format negotiation
slide-18
SLIDE 18

Forewords Pipeline Audio output Video output End

Common problems

  • confusing total latency and buffer usage (or no timing infos

whatsoever),

  • no (glitch-free) pause/resume
  • no explicit drain and/or flush operations
  • device-wide volume controls only
  • no channels layout
  • missing or broken device management
slide-19
SLIDE 19

Forewords Pipeline Audio output Video output End

JACK

  • specific constraints:
  • low latency
  • manually routing
  • always single precision
  • works around most of the requirements
  • not adequate for general use
slide-20
SLIDE 20

Forewords Pipeline Audio output Video output End

ALSA - channels

  • channels count
  • hardware cap, not physical speakers setup
  • plug plugin drops extra channels silently
slide-21
SLIDE 21

Forewords Pipeline Audio output Video output End

ALSA - channels

  • channels count
  • hardware cap, not physical speakers setup
  • plug plugin drops extra channels silently
  • channels map
  • not until recently (and not all drivers)
  • also not necessarily wired speakers
  • SW defaults to stereo with per-app knobs
  • digital output: similar problems
slide-22
SLIDE 22

Forewords Pipeline Audio output Video output End

ALSA (cont’d)

  • no stream volume

(and HW volume controls a big unabstracted mess)

slide-23
SLIDE 23

Forewords Pipeline Audio output Video output End

ALSA (cont’d)

  • no stream volume

(and HW volume controls a big unabstracted mess)

  • defective device management:
  • no hot plug/unplug events (and no udev integration)
  • confused channels and outputs
slide-24
SLIDE 24

Forewords Pipeline Audio output Video output End

OSS

  • Questionable API design (ioctl)
  • Most outstanding functional issues fixed in version 4.
  • Mostly dead: last version 4.2 in 2010
slide-25
SLIDE 25

Forewords Pipeline Audio output Video output End

sndio

  • Not Invented Here syndrome from OpenBSD
  • also RoarAudio server on Linux (almost dead)
  • each and every possible mistake
slide-26
SLIDE 26

Forewords Pipeline Audio output Video output End

sndio

  • Not Invented Here syndrome from OpenBSD
  • also RoarAudio server on Linux (almost dead)
  • each and every possible mistake
  • OK, except per-stream volume
slide-27
SLIDE 27

Forewords Pipeline Audio output Video output End

PulseAudio

  • Decent and well documented

(seems to borrow from Windows Vista)

  • Some bugs, maintainance handed over poorly
  • Bonuses: live fail-over, stream meta infos
slide-28
SLIDE 28

Forewords Pipeline Audio output Video output End

Overall

  • Low-latency and manual setup: JACK
  • Embedded: ALSA (without plugins)
  • All else: PulseAudio (I wish)
slide-29
SLIDE 29

Forewords Pipeline Audio output Video output End

Outline

Pipeline Audio output Video output

slide-30
SLIDE 30

Forewords Pipeline Audio output Video output End

Requirements

  • YCbCr colour space
  • colour subsampling
  • more than 8-bits per component (very near future)
  • planar picture formar
  • scaling
  • blending (subs, overlay)
slide-31
SLIDE 31

Forewords Pipeline Audio output Video output End

Goodies

  • filtering
  • deinterlacing
  • gamma correction
  • noise reduction (nice to have)
  • hardware decoding acceleration and pass-through?
slide-32
SLIDE 32

Forewords Pipeline Audio output Video output End

Video output with X11

  • base (+ MIT-SHM)
  • XVideo extension (+ MIT-SHM)
  • GLX extension
  • Render extension
  • VDPAU-X11
  • VA-X11
  • EGL-X11
slide-33
SLIDE 33

Forewords Pipeline Audio output Video output End

Video output with Wayland

  • base
  • XVideo
  • (Wayland-EGL)
  • Wayland scaler (wlscaler)
  • VDPAU
  • VA-Wayland
  • Wayland-EGL
slide-34
SLIDE 34

Forewords Pipeline Audio output Video output End

XVideo extension

  • originally meant for dedicated hardware overlay
  • no compositing, no blending
  • except with pixmaps support
  • inconsistent cropping
  • provided for backward compability
  • overdue for deprecation
slide-35
SLIDE 35

Forewords Pipeline Audio output Video output End

Renger extension

  • roughly equivalent to wlscaler (but more boilerplate)
  • only RGB
  • only 8-bits per component
slide-36
SLIDE 36

Forewords Pipeline Audio output Video output End

DRM

  • X11, Wayland, headless
  • hardware-dependant
  • not provided by, err, some drivers
  • intended for GL (and VA), not for applications (say

Wayland/Weston developers)

slide-37
SLIDE 37

Forewords Pipeline Audio output Video output End

VDPAU & VA

  • not vendor-neutral
  • VDPAU: AMD(Mesa), NVIDIA, Nouveau(Mesa)
  • VA: Intel OSC
  • XvBA: AMD(Catalyst)
  • high-depth coming?
  • vvvv ??
slide-38
SLIDE 38

Forewords Pipeline Audio output Video output End

OpenGL

  • versions and extensions hell
  • shaders for colour space convesion
  • supports high-depth
  • code reuse on other platforms
  • interoperable with VDPAU
slide-39
SLIDE 39

Forewords Pipeline Audio output Video output End

Overall

  • VDPAU or VA where applicable especially

hardware-accelerated decoding

  • OpenGL
  • buggy drivers...
slide-40
SLIDE 40

Forewords Pipeline Audio output Video output End

Thanks to VideoLAN for sponsporing most of the costs.

Any questions?