Shifting Media app development into high gear Using virtual drivers - - PowerPoint PPT Presentation

▶
shifting media app development into high gear
SMART_READER_LITE
LIVE PREVIEW

Shifting Media app development into high gear Using virtual drivers - - PowerPoint PPT Presentation

Dec 11, 2022 259 likes •757 views

Shifting Media app development into high gear Using virtual drivers to speed up development Helen Koike helen.koike@collabora.com Open First Creative Commons Attribution-ShareAlike 4.0 Summary Classic V4L2 API Vivid Driver Media

slide-1
SLIDE 1

Open First

Shifting Media app development into high gear

Using virtual drivers to speed up development

Helen Koike helen.koike@collabora.com

Creative Commons Attribution-ShareAlike 4.0

slide-2
SLIDE 2
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 2

Summary

slide-3
SLIDE 3
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 3

Summary

slide-4
SLIDE 4

ELC 2018 4

slide-5
SLIDE 5

ELC 2018 5

Sensor Sepia Greyscale Composer Scaler

slide-6
SLIDE 6

ELC 2018 6

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------

APP

  • User

space Kernel space Physical device Driver

slide-7
SLIDE 7

ELC 2018 7

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device

Driver

Video stream Read/Write/MMAP APP

slide-8
SLIDE 8

ELC 2018 8

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device Video stream General confgs * Img fmt * Bufers * Video std * Frame rate . . Standard Controls * Contrast * Brightness * Gamma . . . Custom Controls * DRV DEF 1 * DRV DEF 2 * DRV DEF 3 . . . Read/Write/MMAP IOCTLs

Driver

APP

slide-9
SLIDE 9

ELC 2018 9

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device Driver APP

slide-10
SLIDE 10

ELC 2018 10

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device Driver APP

slide-11
SLIDE 11

ELC 2018 11

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device The Virtual Video T est Driver (VIVID) APP

slide-12
SLIDE 12

ELC 2018 12

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device The Virtual Video T est Driver (VIVID) APP

slide-13
SLIDE 13

Vivid driver

ELC 2018 13

slide-14
SLIDE 14
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 14

Summary

slide-15
SLIDE 15
  • Embedded systems: great variety of devices
  • Hardware complexity
  • Similar confguration for diferent entities

ELC 2018 15

Media API Motivation

slide-16
SLIDE 16

ELC 2018 16

Sensor Sepia Greyscale Composer Scaler contrast contrast

slide-17
SLIDE 17

ELC 2018 17

  • ------------------------------------------------------- /dev/video* --------------------------------------------------------
  • User

space Kernel space Physical device Video stream General confgs * Img fmt * Bufers * Video std * Frame rate . . Standard Controls * Contrast * Brightness * Gamma . . . Custom Controls * DRV DEF 1 * DRV DEF 2 * DRV DEF 3 . . . Read/Write/MMAP IOCTLs

Driver

APP

slide-18
SLIDE 18

ELC 2018 18

Sensor Sepia Greyscale Composer Scaler

slide-19
SLIDE 19

ELC 2018 19

Sensor Sepia Greyscale Composer Scaler

slide-20
SLIDE 20

ELC 2018 20

Sensor Sepia Greyscale Composer Scaler /dev/video0 /dev/v4l-subdev0 /dev/v4l-subdev1 /dev/v4l-subdev2

slide-21
SLIDE 21

ELC 2018 21

/dev/video0 R/W/MMAP IOCTLs APP /dev/v4l- subdev0

IOCTLs APP

/dev/v4l- subdev1

IOCTLs APP

/dev/v4l- subdev2

IOCTLs APP

Sepia contrast Greyscale contrast Scaler

Subdev API

slide-22
SLIDE 22

ELC 2018 22

/dev/v4l-subdev0 /dev/v4l-subdev14 /dev/video0 /dev/video42 /dev/v4l-subdev98 /dev/video2 /dev/v4l-subdev6 /dev/v4l-subdev21 /dev/v4l-subdev11 /dev/v4l-subdev31 APP ? ? ? ? ? ? ? ? ? ? ? ?

What is part of the device? What is the topology?

slide-23
SLIDE 23

ELC 2018 23

Sensor Sepia Greyscale Composer Scaler /dev/video0 /dev/v4l-subdev0 /dev/v4l-subdev1 /dev/v4l-subdev2

slide-24
SLIDE 24

ELC 2018 24

Sensor Sepia Greyscale Composer Scaler

  • ut

/dev/video0 /dev/v4l-subdev0 /dev/v4l-subdev1 /dev/v4l-subdev2

slide-25
SLIDE 25

ELC 2018 25

/dev/video0 R/W/MMAP IOCTLs APP /dev/v4l- subdev0

IOCTLs APP

/dev/v4l- subdev1

IOCTLs APP

/dev/v4l- subdev2

IOCTLs APP

Sepia contrast Greyscale contrast Scaler

Media API

/dev/media0

IOCTLs APP

slide-26
SLIDE 26

ELC 2018 26

Sensor Sepia Greyscale Composer Scaler

  • ut

/dev/video0 /dev/v4l-subdev0 /dev/v4l-subdev1 /dev/v4l-subdev2 /dev/media0

IOCTLs APP

slide-27
SLIDE 27

ELC 2018 27

Sensor Sepia Greyscale Composer Scaler

  • ut

/dev/video0 /dev/v4l-subdev0 /dev/v4l-subdev1 /dev/v4l-subdev2 /dev/media0

IOCTLs APP

slide-28
SLIDE 28

ELC 2018 28

  • ------------------------------------------------------- /dev/media* --------------------------------------------------------
  • User

space Kernel space Physical device Driver APP

slide-29
SLIDE 29

ELC 2018 29

  • ------------------------------------------------------- /dev/media* --------------------------------------------------------
  • User

space Kernel space Physical device Driver APP

slide-30
SLIDE 30

ELC 2018 30

  • ------------------------------------------------------- /dev/media* --------------------------------------------------------
  • User

space Kernel space Physical device The Virtual Media Controller Driver (VIMC) APP

slide-31
SLIDE 31

ELC 2018 31

  • ------------------------------------------------------- /dev/media* --------------------------------------------------------
  • User

space Kernel space Physical device The Virtual Media Controller Driver (VIMC) APP ?

slide-32
SLIDE 32
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 32

Summary

slide-33
SLIDE 33

ELC 2018

  • -------------- VIMC Confgfs API --------------------- /dev/media0 --------------------- /dev/media1 -----------

User space Kernel space APP Developer create create

33

slide-34
SLIDE 34

ELC 2018 34

Vimc: submodules

Subdev1.ko Subdev2.ko SubdevN.ko Implementation of subdevice drivers are modularized and doesn’t need to alter Vimc’s core code.

slide-35
SLIDE 35

ELC 2018 35

vimc-sensor.ko vimc-scaler.ko vimc-debayer.ko vimc-capture.ko ... custom-sd1.ko custom-sd2-ko custom-sd3.ko ... Standard Manufacturer

APP Developer

Emulate diferent topologies to test the app in several scenarios

slide-36
SLIDE 36
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 36

Summary

slide-37
SLIDE 37

Vimc Driver

  • Proposed by Laurent Pinchart for Outreachy in 2015
  • Merged in Kernel 4.12
  • Moving slowly

ELC 2018 37

slide-38
SLIDE 38

Vimc Driver: current state

  • Basic standard submodules:

– vimc-capture.ko – vimc-sensor.ko – vimc-debayer.ko – vimc-scaler.ko

  • Hard-coded topology (re-compilation required)

ELC 2018 38

slide-39
SLIDE 39

ELC 2018 39

slide-40
SLIDE 40

ELC 2018 40

Vimc Driver

slide-41
SLIDE 41

Userspace tools

  • v4l-utils

media-ctl -d /dev/media0 --print-dot > /tmp/out.dot && xdot /tmp/out.dot media-ctl -d /dev/media0 -V "'Sensor A':0[fmt:RGB888_1X24/600x600]" media-ctl -v -d /dev/media0 --links "'Debayer A':1->'Scaler':0 [0]"

  • Yavta (Yet Another V4L2 T

est Application)

yavta --format RGB24 --size 600x600 /dev/video0

ELC 2018 41

slide-42
SLIDE 42
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 42

Summary

slide-43
SLIDE 43

ELC 2018 43

‘Debayer A’ vimc-debayer.ko

1

‘Sensor A’ vimc-sensor.ko ‘Raw Capture 0’ vimc-capture.ko

  • Entities

– Name – Submodule

  • Pads

– Source – Sink

  • Links

Confgfs API: Topology

slide-44
SLIDE 44

ELC 2018 44

mkdir “MEDIA_NAME”

slide-45
SLIDE 45

ELC 2018 45

‘Debayer A’ vimc-debayer.ko

1

‘Sensor A’ vimc-sensor.ko ‘Raw Capture 0’ vimc-capture.ko

mkdir “SUBMOD:NAME”

slide-46
SLIDE 46

ELC 2018 46

‘Debayer A’ vimc-debayer.ko

1

‘Sensor A’ vimc-sensor.ko ‘Raw Capture 0’ vimc-capture.ko

mkdir “NAME1:PAD->NAME2:PAD”

slide-47
SLIDE 47
  • Classic V4L2 API

→ Vivid Driver

  • Media API (extension)

→ Vimc Driver

  • Vimc:
  • Submodules
  • Current state
  • Confgfs API
  • Future work

ELC 2018 47

Summary

slide-48
SLIDE 48

Vimc Driver: future work

  • API in Confgfs (WIP)
  • vimc-input.ko (WIP)
  • Optimizations of img processing in the pipeline (WIP)
  • GUI User space tool
  • More standard submodules
  • Add more V4L2 mechanisms / controls / options

ELC 2018 48

slide-49
SLIDE 49

Thank you!

Helen Koike helen.koike@collabora.com

This work is licensed under https://creativecommons.org/licenses/by-sa/4.0/