v4l2 stream sharing Brandon Philips brandon@ifup.org - - PowerPoint PPT Presentation

▶
v4l2 stream sharing
SMART_READER_LITE
LIVE PREVIEW

v4l2 stream sharing Brandon Philips brandon@ifup.org - - PowerPoint PPT Presentation

Oct 15, 2023 239 likes •573 views

v4l2 stream sharing Brandon Philips brandon@ifup.org brandon@suse.com Motivations Single Streaming Application Problem 3 Motivations User-space autofocus 4 Motivations User-space autofocus Face tracking and panning 5

slide-1
SLIDE 1

v4l2 stream sharing

Brandon Philips brandon@ifup.org brandon@suse.com

slide-2
SLIDE 2

Motivations

slide-3
SLIDE 3

3

Single Streaming Application Problem

slide-4
SLIDE 4

4

Motivations

  • User-space autofocus
slide-5
SLIDE 5

5

Motivations

  • User-space autofocus
  • Face tracking and panning
slide-6
SLIDE 6
slide-7
SLIDE 7

7

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
slide-8
SLIDE 8

8

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
  • Backup video going to Skype/Pidgin
slide-9
SLIDE 9

9

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
  • Backup video going to Skype/Pidgin
  • Only tranform stream once
slide-10
SLIDE 10

Approach

slide-11
SLIDE 11

11

Approach

  • Per user session video daemon
  • Applications share using RO shm area
  • Communicate with daemon via sockets
  • Sound familiar?
  • I have bene reading PulseAudio ;)
slide-12
SLIDE 12

12

Help Wanted: AnekVideo

github.com/philips/anekvideo git://ifup.org/philip/anekvideo.git

slide-13
SLIDE 13

13

One memcpy too much

slide-14
SLIDE 14

14

Multi-map v4l2

  • Modify v4l2 to allow multiple buffer users
  • First process is “authorized” to make format changes
  • REQBUF increments number of stream users
  • QBUF increments refcount on buffer
  • DQBUF decrements refcount on buffer
slide-15
SLIDE 15

15

Multi-map v4l2 cont

  • Will need to be implemented per driver
  • Multi map will benefit from a daemon
  • Format negotiation to satisfy all clients instead of first
  • Manage applications that fall behind
  • v4l2 userptr memory type to copy directly to SHM
slide-16
SLIDE 16

Questions & Thanks

slide-17
SLIDE 17

v4l2 stream sharing

Brandon Philips brandon@ifup.org brandon@suse.com

This work isn't sponsored by Novell but they are my employer and it is a fun place to work. Plus, this openSUSE template looks awesome compared to the OO.org defaults :D

slide-18
SLIDE 18

Motivations

slide-19
SLIDE 19 3

Single Streaming Application Problem

Click to add text

slide-20
SLIDE 20 4

Motivations

  • User-space autofocus

A number of cameras now only do manual focus and rely on the OS to do focus for

  • them. This requires a daemon that can look at the incoming frames and do the

focusing

slide-21
SLIDE 21 5

Motivations

  • User-space autofocus
  • Face tracking and panning

A number of cameras on other OSes offer drivers that will detect faces and crop the frame to that face region. This is useful for Skype and other video conferencing applications that have very small frame sizes

slide-22
SLIDE 22
slide-23
SLIDE 23 7

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
slide-24
SLIDE 24 8

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
  • Backup video going to Skype/Pidgin

It would be nice to be able to save the session going over Skype/Pidgin/etc. I talk to my future inlaws and sometimes practice hindi and it would be nice to see how poorly my speaking was.

slide-25
SLIDE 25 9

Motivations

  • User-space autofocus
  • Face tracking and panning
  • “Take photo of thief” software
  • Backup video going to Skype/Pidgin
  • Only tranform stream once

Some devices offer driver specific frame formats that libv4l converts. It would be nice to not make this conversion twice.

slide-26
SLIDE 26

Approach

slide-27
SLIDE 27 11

Approach

  • Per user session video daemon
  • Applications share using RO shm area
  • Communicate with daemon via sockets
  • Sound familiar?
  • I have bene reading PulseAudio ;)
slide-28
SLIDE 28 12

Help Wanted: AnekVideo

Click to add text

github.com/philips/anekvideo git://ifup.org/philip/anekvideo.git

This project has been relegated to my second hobby project status since last years

  • talk. I have regained interest though in the last month or so.

Having a group of motivated people helping would be a better situation then me working alone though.

slide-29
SLIDE 29 13

One memcpy too much

Click to add text

This is the first USB 3.0 webcam to be announced. It is from Point Grey and streams “uncompressed” 1920x1080p frames at 60fps! Originally I envisioned AnekVideo would copy and share frames via the shm in every case.

slide-30
SLIDE 30 14

Multi-map v4l2

  • Modify v4l2 to allow multiple buffer users
  • First process is “authorized” to make format changes
  • REQBUF increments number of stream users
  • QBUF increments refcount on buffer
  • DQBUF decrements refcount on buffer
slide-31
SLIDE 31 15

Multi-map v4l2 cont

  • Will need to be implemented per driver
  • Multi map will benefit from a daemon
  • Format negotiation to satisfy all clients instead of first
  • Manage applications that fall behind
  • v4l2 userptr memory type to copy directly to SHM
slide-32
SLIDE 32

Questions & Thanks