Gstreamer Editing Services Video Editing in your pocket (size of - - PowerPoint PPT Presentation

▶
gstreamer editing services video editing in your pocket
SMART_READER_LITE
LIVE PREVIEW

Gstreamer Editing Services Video Editing in your pocket (size of - - PowerPoint PPT Presentation

Feb 10, 2023 121 likes •371 views

Gstreamer Editing Services Video Editing in your pocket (size of pocket not specified) Edward Hervey edward.hervey@collabora.co.uk Edward Hervey Co-founder of Collabora Multimedia FLOSS user since 1995 GStreamer Hacker since 2003

slide-1
SLIDE 1

Gstreamer Editing Services Video Editing in your pocket

(size of pocket not specified)

Edward Hervey edward.hervey@collabora.co.uk

slide-2
SLIDE 2

Edward Hervey

  • Co-founder of Collabora Multimedia
  • FLOSS user since 1995
  • GStreamer Hacker since 2003
  • PiTiVi video editor
  • French (despite not striking)
slide-3
SLIDE 3

Primary goal of GES project

  • Provide everything needed to make

editing applications trivial to write

  • GNonLin/GStreamer were not enough
  • Take GES + GStreamer
  • Sprinkle UI on top of it
  • => You have an editor !
slide-4
SLIDE 4

Secondary Goals

  • Complete high-level solution

– Not just editing, but also playback,

encoding, media discovery, …

  • Flexible

– Not just one use-case in mind

  • NO HACKS !

– Upstream as much as possible

slide-5
SLIDE 5

Breakdown

  • GES high-level library

– Timeline, Layer, Track – Convenience objects

  • Peripheral libraries/improvements

– libgstprofile, encodebin – GstDiscoverer – …

  • Lessons learnt
  • Ideas and improvements
slide-6
SLIDE 6

Gstreamer Editing Services

  • Funded by Nokia
  • LGPL
  • 12 KLOC
  • C/GObject, based on GStreamer and

GnonLin.

  • Examples, unit tests, API documentation
  • High-level API
  • Brandon Lewis co-developer
  • Meego 1.3
  • git.collabora.co.uk user/edward/gst-

editing-services

slide-7
SLIDE 7

GESTimeline

  • Central object
  • Controls Layers and Tracks
  • Is a GstBin
  • Save/Load
slide-8
SLIDE 8

GESTimeline

slide-9
SLIDE 9

GESTimelinePipeline

  • GstPipeline (like playbin2)
  • Takes a GESTimeline
  • Playback/Preview (autosinks)
  • Rendering (encodebin)
  • Thumbnailing/Screenshot
slide-10
SLIDE 10

GESTimelineLayer

  • Takes «natural» objects

– Files (Video, Audio, Images,..) – Transitions – Credits, Titles – …

  • Most user-centric part of the Timeline
  • Can add more (ex: above:overlay,

under:soundtrack)

  • Media-agnostic
slide-11
SLIDE 11

GESTimelineObject

  • Basic properties

– Start (When does it go ?) – Duration (For how long ?) – In-point (offset in the object) – Priority (Precedence over other

  • bjects)
  • Creates and controls Track object(s)
  • Base classes for Sources, Transitions,

Overlays

  • Create your own TimelineObject

– Templates

slide-12
SLIDE 12

Available TimelineObjects

  • GESTimelineFileSource

– Video, Audio, Picture... – Will figure out duration on its own

  • GESTransition

– Crossfade, most SMPTE transitions – Audio also

slide-13
SLIDE 13

GESSimpleTimelineLayer

  • List-based API
  • Only care about the ordering and duration
  • Takes care of adjusting the time position
  • f all objects
slide-14
SLIDE 14

GESTrack

  • One per media output (Audio, Video,

Subtitle, …)

  • Control what media is outputted

– Raw Audio/Video... – … or already encoded data

  • Only set the Track(s) you want on the

Timeline

– Ex : video-only render/playback

slide-15
SLIDE 15

Layer/Track interaction

slide-16
SLIDE 16

GESTrackObject

  • Produce/Modify the media
  • GnlObject under the hood
  • Essentially a GstBin

– You can put anything you want in it

slide-17
SLIDE 17

GESFormatter

  • Timeline load/save (serialization)
  • Create your own subclass
slide-18
SLIDE 18

GstDiscoverer

  • Get information about a URI
  • Audio ? Video ?
  • Duration ?
  • Tags ?
  • Codec ? Media properties (width/height...)
  • Gst-plugins-base 0.10.31
  • Used by GES if needed
slide-19
SLIDE 19

libgstprofile and encodebin

  • Make rendering as easy as playback
  • Long standing problem
  • GstEncodingProfile

– Describe streams and not elements

  • Encodebin element

– (dynamic) sink pads based on profile – Can do passthrough – Conversion elements

  • Proposed for gst-plugins-base
  • Bugzilla #627476
slide-20
SLIDE 20

gst_video_convert_frame

  • Convert a video GstBuffer to any format
  • Backported from playback plugin
  • Added encoding capabilities (to images)
  • GstBuffer* gst_video_convert_frame(

GstBuffer *buf, const GstCaps *to_caps, GstClockTime timeout, GError **error)

  • Gst-plugins-base 0.10.31
slide-21
SLIDE 21

GstElementFactoryList

  • Backported and improved from playback

plugin

  • « Get all factories of a certain <type> [,

that can handle <media> [, in a certain <direction> ]] »

– Ex : Available video fx, encoders,...

  • Gstreamer core 0.10.31
slide-22
SLIDE 22

Lessons learnt

  • Codecs:

– Not the obvious cpu bottleneck – GstSegment handling is not an

  • ption
  • Elements:

– QoS for best end-user experience

  • Editing brings complex pipelines !

– Optimisation in core (caps nego,

data passing, …)

  • Avoid memcpy (videoscale add-border)
  • ORC ORC ORC !
slide-23
SLIDE 23

Ideas / Improvements

  • GnlComposition scheduling mode

– Always ready (high mem, lowest lat) – Neighbour ready (med mem, med lat

) + Using QoS

– On demand (lowest mem, high lat)

  • Single instance HW accelerated decoders
  • pre-render/cache on-demand
  • Proxy support

– Of complex operations, sources...

and timelines

slide-24
SLIDE 24

Thankyou ! Any Questions ?