High complexity GStreamer pipelines Buzztard / Audio / Gstreamer - PowerPoint PPT Presentation

High complexity GStreamer pipelines Buzztard / Audio / Gstreamer Stefan Sauer <ensonic@hora-obscura.de>, 24.10.2011

What is buzztard? What is buzztard? ?

NO

NO

We're getting We're getting there there

Short history Short history ● Buzz: free, but closed source and windows only ● Dead in 2002, continued in 2009

● Buzztard: open source, focuses on Linux ● Since 2003 ● Uses GStreamer :)

Foundation: Registry Foundation: Registry ● Book keeping of features ● Fast startup (cache) ● Safe plugin probing (out-of-process)

Foundation: Scalability Foundation: Scalability ● Huge pipelines ● >500 elements ● >50 threads ● Example buzztard song of this size ● ~ 50 CPU use on a 1.6 GHz Intel Atom ● on a gstreamer debug enabled build

Foundation: Sequencing Foundation: Sequencing ● GstController ● Sync control-changes with rendering ● Outside of ui threads

Foundation: Sparse streams Foundation: Sparse streams ● Gap flag on buffers ● elements take shortcuts on silence

Foundation: DecodeBin(2) Foundation: DecodeBin(2) ● Handling of different audio formats is a NO-OP VORBIS MP3 WAV AIFF AAC AU

Foundation : Audio IO Foundation : Audio IO ● Everything is there! ● alsa, jack, pulseaudio, ...

Foundation: Elements Foundation: Elements ● Lots of utility elements available ● data conversion, level meter, spectrum analyzer, ...

Foundation: Introspection Foundation: Introspection ● Lots of GObject goodness ● Classify elements ● Generate dialogs

Tricky: trick modes Tricky: trick modes ● Jog dial like scrubbing ● All elements need to play to the rules ● Adder fixed in git HEAD ● Still occational lockups in reverse mode

Tricky: seamless loops Tricky: seamless loops ● Don't miss a beat ● Adder fixed in git HEAD

Tricky: dynamic linking Tricky: dynamic linking ● Add/remove elements while playing ● Fragile, complicated and under-documented

Tricky: dynamic linking Tricky: dynamic linking ● data flow on disconnected pads is fatal => pad-blocking ● pad-blocking on inactive pads hangs ● Pad-blocking is async ● Happens on data-flow over the pad ● Mind the direction: src for push, sink for pull ● Block the the the last src pad for push, first sink pad for pull ● Need to emulate startup/shutdown

Tricky: dynamic linking Tricky: dynamic linking ● What to do with conditional elements ● un-parent from the pipeline ● Keep in the pipeline and lock the state

Tricky: dynamic linking Tricky: dynamic linking ● Add elements ● Activate pads ● Block pads ● Change state to PAUSED ● Link ● Seek (the flush unblocks) ● Unlock the state ● Change state to PLAYING

Tricky: dynamic linking : add 1 Tricky: dynamic linking : add 1

Tricky: dynamic linking : add 2 Tricky: dynamic linking : add 2

Tricky: dynamic linking : add 3 Tricky: dynamic linking : add 3

Tricky: dynamic linking Tricky: dynamic linking ● Remove elements ● Block pads ● change state to NULL ● Lock the state ● Unlink ● Unblock pads

Tricky: dynamic linking : remove Tricky: dynamic linking : remove

Tricky: dynamic linking Tricky: dynamic linking ● Buzztard svn repo: design/gst/dynlink{2,3}.c

WIP: plugin wrappers WIP: plugin wrappers ● Ladspa and lv2 still have issues (sources) ● More wrappers needed (dssi, vst)

WIP: low latency audio WIP: low latency audio ● Determine effective latency in push mode ● Make pull mode work for audio ● It does not lockup anymore ● Queues need changes to not buffer

WIP: push mode latency WIP: push mode latency ● Latency related to bpm/tpb and sampling rate ● e.g.: chunk-size for sink 120000 µs = 120 ms

WIP: push mode latency WIP: push mode latency ● 140 ms delay ~ 46ms per queue

Future Future ● GStreamer 1.0 :) ● EncodeBin for recording ● discoverer in the wave-table browser ● if the gtk+filechooser could be populated with disco- info asynchronously … ● tempo information as sticky events on the pads ● beats-per-minute + ticks-per-beat

Thanks! Questions?

Thanks Thanks sheetmusic: by 'starrise' Trey Jones ● sequencer: shows propellerheads reason ● icons are from gnome and tango icon themes ●