Cameras in embedded systems: Device tree and ACPI view Sakari Ailus - - PowerPoint PPT Presentation

▶
cameras in embedded systems device tree and acpi view
SMART_READER_LITE
LIVE PREVIEW

Cameras in embedded systems: Device tree and ACPI view Sakari Ailus - - PowerPoint PPT Presentation

Aug 11, 2023 49 likes •439 views

Cameras in embedded systems: Device tree and ACPI view Sakari Ailus - Intel A typical embedded system with a camera Image Signal Processor Raw camera sensor Lens voice coil vana camera regulator module vdig lens GPIO reset

slide-1
SLIDE 1

Cameras in embedded systems: Device tree and ACPI view

Sakari Ailus - Intel

slide-2
SLIDE 2

A typical embedded system with a camera

SoC camera module lens voice coil sensor lens voice coil lens ISP CSI-2 I²C I²C GPIO regulator reset vana vdig clock

  • Image Signal Processor
  • Raw camera sensor
  • Lens voice coil
slide-3
SLIDE 3

Raw sensors

  • Raw sensors have little processing logic in the

sensor itself

– Analogue and digital gain but not much more

This is how white looks like! -->

slide-4
SLIDE 4

Image signal processors

  • Process the image

for viewing

After ISP processing white looks like this --->

slide-5
SLIDE 5

Video4Linux and Media controller

  • Video4Linux (V4L2) is the Linux API for capturing

images

– Video capture cards – USB webcams – Cameras in embedded devices

  • Media controller is a control interface for complex

media devices

– Image pipeline discovery and configuration – Device discovery

slide-6
SLIDE 6

Example of a media graph

ISP sensor

slide-7
SLIDE 7

Probing

  • Each driver is

probed separately

  • How to tell drivers they all are part of the same

media device?

PCI, AMBA etc. ISP I2C controller sensor lens voice coil

slide-8
SLIDE 8

Media device setup

1.media_device_init() 2.v4l2_device_register() 3.video_register_device() 5.v4l2_device_register_subd ev(isp) 6.v4l2_register_subdev_nod es() 7.media_device_register()

4.v4l2_device_register_subdev( sensor)

Media device driver sensor driver

slide-9
SLIDE 9

V4L2 async

slide-10
SLIDE 10

V4L2 async

  • The V4L2 async framework facilitates sub-

device registration

  • V4L2 sub-device device node creation and

media device registration postponed after probe

  • To do its job, the V4L2 async framework makes

use of firmware provided information

slide-11
SLIDE 11

V4L2 async example (ISP)

device base ISP driver ISP DT node sensor DT node V4L2 async async sub-device li st async notifjer list probe(dev) parse local endpoints
  • f_graph_get_remote_port_parent(endpoint)
v4l2_async_register_notifjer(notifjer) Look for sub-devices mat ching the notifjer list No match found Add ISP notifjer
slide-12
SLIDE 12

V4L2 async (sensor)

dev ice base ISP driver sensor driver sensor DT node V4L2 async async notifjer list probe(dev) v4l2_of_parse_endpoint() endpoint properties v4l2_async_register_subdev(sd) Look for a notifjer matchi n the sub-device Found it! notifjer->bound() notifjer->complete()
slide-13
SLIDE 13

Device tree

slide-14
SLIDE 14

Device tree

  • System hardware description in a human readable format
  • Originates from Sparc / Open Firmware
  • Primarily used on embedded systems

– ARM – Also PowerPC, Sparc and x86

  • Tree structure

– Nodes – Properties

  • Source code compiled into binary before use
slide-15
SLIDE 15

Device Tree standard and bindings

  • Device Tree specification maintained by

devicetree.org

– Syntax and some semantics

  • Bindings define the interface between the firmware

and the software

– Bindings are Operating System specific – Linux Device tree binding documentation part of the Linux

kernel source

– FreeBSD developers appear to be converging towards

using Linux DT bindings

slide-16
SLIDE 16

Device tree graphs

  • phandle properties can be used to refer to
  • ther nodes in the tree
  • Port is an interface in a device (as in an IP

block)

  • Endpoint describes one end of a connection to

a port [7]

slide-17
SLIDE 17

Sensor node

&i2c2 { smia_1: camera@10 { compatible = "nokia,smia"; reg = <0x10>; /* No reset gpio */ vana-supply = <&vaux3>; clocks = <&isp 0>; clock-frequency = <9600000>; nokia,nvm-size = <(16 * 64)>; port { smia_1_1: endpoint { link-frequencies = /bits/ 64 <199200000 210000000 499200000>; clock-lanes = <0>; data-lanes = <1 2>; remote-endpoint = <&csi2a_ep>; }; }; }; }; source: arch/arm/boot/dts/omap3-n9.dts

slide-18
SLIDE 18

ISP node board specific part

&isp { vdd-csiphy1-supply = <&vaux2>; vdd-csiphy2-supply = <&vaux2>; ports { port@2 { reg = <2>; csi2a_ep: endpoint { remote-endpoint = <&smia_1_1>; clock-lanes = <2>; data-lanes = <1 3>; crc = <1>; lane-polarities = <1 1 1>; }; }; }; }; source: arch/arm/boot/dts/omap3-n9.dts

slide-19
SLIDE 19

OF graph API

  • Parse port and endpoint nodes under device

nodes

  • Enumerate over endpoints
  • Obtain remote endpoint

– Based on the phandle value

slide-20
SLIDE 20

ACPI

slide-21
SLIDE 21

ACPI

  • Advanced Configuration and Power Interface
  • Operating system independent
  • Origins in x86 and PC

– Increasingly used in embedded systems

  • Device discovery and enumeration
  • Power management
  • ACPI methods

– Runnable code – ACPI virtual machine

slide-22
SLIDE 22

ACPI

  • ACPI specifications developed by UEFI Forum

– Roughly one specification per year

  • What do you do if you need to add a new kind
  • f a device?

– A new ACPI specification?

slide-23
SLIDE 23

ACPI Device Specific Data

  • _DSD object type part of ACPI 5.1 and later

– Key-value pairs (property extension) and – Tree structures (hierarchical data extension)

  • Together property and data extension could be

used to implement very similar functionality to Devicetree

  • _DSD property registry [6]

– Light-weight approach for registering _DSD properties

slide-24
SLIDE 24

fwnode property API

  • Access properties independently of underlying

firmware implementation

– Device Tree – ACPI

  • Makes use of ACPI _DSD property extension

[2]

slide-25
SLIDE 25

Future work

slide-26
SLIDE 26

Fwnode graph API

  • Functionally the same as the OF graph API

– But is firmware independent

  • Device tree implementation is used on Device

tree

  • Makes use of the _DSD hierarchical data

extension [3] on ACPI

  • Implementation at RFC level [4]
slide-27
SLIDE 27

V4L2 fwnode API

  • ”V4L2 ACPI support”

– Embedded systems with I²C components – Requires both fwnode graph API and V4L2 fwnode API

  • Same functionality as V4L2 OF API
  • V4L2 fwnode and V4L2 OF fully interoperable

– Sub-device driver using V4L2 fwnode works with a media

device driver using V4L2 OF

– and vice versa!

  • RFC implementation available [5]
slide-28
SLIDE 28

Flash

  • LED flash devices supported
  • But the kernel has no knowedge which sensor

they're related to

– This is rather important if there are multiple

cameras in the system, such as most mobile phones nowadays

  • Standardise phandle property for this?
slide-29
SLIDE 29

Camera module

  • Currently there's no ”camera module” concept in

the kernel (nor DT or ACPI)

  • Camera module construction is important for the

user space

– Which sensor and lens are related? – What kind of lens is there? – What's the voice coil spring constant? – Is there an infra red filter? What kind of filter is it? – What's the aperture size?

slide-30
SLIDE 30

Camera module power on and power off sequences

  • Regulators, clocks and / or GPIOs may be shared between

module components

  • Power on and power off sequences device component

specific

– Which order and when each resource may be enabled? – E.g. regulator and clock are enabled, then after 10 ms the reset

GPIO can be lifted and the device is ready for use

  • Requirements of both

lens and sensor must be considered for module power-up sequence

camera module lens voice coil sensor lens voice coil lens CSI-2 I²C reset vana vdig clock

slide-31
SLIDE 31

Questions?

slide-32
SLIDE 32

References

[1] http://www.uefi.org/acpi [2] http://www.uefi.org/sites/default/files/resources/_DSD-device-properties-UUID.pdf [3] http://www.uefi.org/sites/default/files/resources/_DSD-hierarchical-data-extension-UUID-v1.pdf [4] http://www.spinics.net/lists/linux-acpi/msg69547.html [5] http://www.spinics.net/lists/linux-media/msg106160.html [6] https://github.com/ahs3/dsd [7] Documentation/devicetree/bindings/graph.txt

slide-33
SLIDE 33

Firmware logistics

Device tree ACPI

BIOS vendor system vendor BIOS in flash memory SoC vendor OS binaries runtime ACPI tables mother- board support website

selective table replacement (initrd)

SoC vendor OS binaries system vendor flash memory Linux kernel dts Device tree

appended to kernel image

slide-34
SLIDE 34

ACPI camera example

slide-35
SLIDE 35

ACPI camera example

Scope (\_SB.PCI0.I2C2) { Device (CAM0) { Name (_DSD, Package () { /* device specific data */ Package () { Package () { "compatible", Package () { "nokia,smia" } }, Package () { "lanes", 4 }, Package () { "clock-frequency", 24000000 }, }, /* data extension */ Package () { Package () { "ports", "PRTS" }, } })

slide-36
SLIDE 36

ACPI camera example

Name (PRTS, Package() { /* data extension */ Package () { Package () { "port@0", "PRT0" }, } }) Name (PRT0, Package() { /* device specific data */ Package () { Package () { "port", 0 }, }, /* data extension */ Package () { Package () { "endpoint@0", "EP0" }, } })

slide-37
SLIDE 37

ACPI camera example

Name (EP0, Package() { /* device specific data */ Package () { Package () { "endpoint", 0 }, Package () { "clock-lanes", 0 }, Package () { "data-lanes", Package () { 1, 2, 3, 4 } }, Package () { "link-frequencies", Package () { 209600000, 342000000, 451200000 } }, Package () { "remote-endpoint", Package() { \_SB.PCI0.ISP, 0, 0, 0 } }, }, }) } }

slide-38
SLIDE 38

ACPI ISP example

Scope (\_SB.PCI0) { Device (ISP) { Name (_DSD, Package () { /* data extension */ Package () { Package () { "ports", "PRTS" }, } }) Name (PRTS, Package() { /* data extension */ Package () { Package () { "port@4", "PRT4" }, } })

slide-39
SLIDE 39

ACPI ISP example

Name (PRT4, Package() { /* device specific data */ Package () { Package () { "port", 4 }, /* CSI-2 port number */ }, /* data extension */ Package () { Package () { "endpoint@0", "EP0" }, } }) Name (EP0, Package() { /* device specific data */ Package () { Package () { "endpoint", 0 }, Package () { "clock-lanes", 0 }, Package () { "data-lanes", Package () { 1, 2, 3, 4 } }, Package () { "remote-endpoint", Package () { \_SB.PCI0.I2C2.CAM0, 0, 0, 0 } }, }, }) } }