Using Linux Media Controller for Wayland/Weston Renderer Technology - - PowerPoint PPT Presentation

▶
using linux media controller for wayland weston renderer
SMART_READER_LITE
LIVE PREVIEW

Using Linux Media Controller for Wayland/Weston Renderer Technology - - PowerPoint PPT Presentation

Jan 14, 2023 145 likes •606 views

Using Linux Media Controller for Wayland/Weston Renderer Technology Consulting Company Research, Development & Takanari Hayama Global Standard taki@igel.co.jp http://www.igel.co.jp/ 1 Technology Consulting Company IGEL Co.,Ltd.

slide-1
SLIDE 1

Technology Consulting Company IGEL Co.,Ltd.

Technology Consulting Company Research, Development & Global Standard

Using Linux Media Controller for Wayland/Weston Renderer

Takanari Hayama taki@igel.co.jp http://www.igel.co.jp/

1

slide-2
SLIDE 2
  • Wayland/Weston Overview
  • Porting Weston to R-Car
  • Why Linux Media Controller Renderer?
  • Linux Media Controller Framework
  • V4L2 Renderer Design
  • Conclusions

2

Agenda

slide-3
SLIDE 3

WAYLAND/WESTON OVERVIEW

3

slide-4
SLIDE 4

Weston Architecture

Wayland Client Weston Wayland Client Compositor Core Compositor (e.g. DRM) Renderer (e.g. GL) Shell (e.g. desktop) Shell Client (e.g. desktop-shell)

Wayland IPC Local API Call Confidential 4

slide-5
SLIDE 5

Weston w/ DRM Backend

5

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES OpenGL/ES compositor-drm gl-renderer shell libgbm/ libdrm compositor- core pixman- renderer libpximan GPU Kernel Driver DRM/KMS Driver Qt, EFL, etc.

Weston Components Wayland Components Other Components IPC API Call

User Space Kernel Space

slide-6
SLIDE 6

wayland protocol

6

Rendering and Composition: Overview (GL-Renderer)

Wayland Client OpenGL/ES + wayland_egl GPU driver for Wayland Weston / DRM Compositor OpenGL/ES + WL Ext. GBM DRM/KMS GPU GPU Driver for Wayland Display

wl_egl_window Type wl_egl_window Type

RENDERING DISPLAYING

wl_buffer Type wl_buffer Type

FRAME BUFFERS

gbm_surface Type

CLIENT BUFFERS

wl_buffer Type

FULL SCREEN

wl_buffer Type DRM/KMS BO Type

SCANOUT BUFFERS

Software Hardware

Wayland/Weston Components Khronos w/ Wayland Ext. OSS Components Hardware Specific

slide-7
SLIDE 7

wayland protocol

7

Rendering and Composition: Window Composition

Wayland Client OpenGL/ES + wayland_egl GPU driver for Wayland Weston / DRM Compositor OpenGL/ES + WL Ext. GBM DRM/KMS GPU GPU driver for Wayland Display

wl_egl_window Type wl_egl_window Type

RENDERING DISPLAYING

wl_buffer Type wl_buffer Type

FRAME BUFFERS

gbm_surface Type

CLIENT BUFFERS

wl_buffer Type

FULL SCREEN

wl_buffer Type DRM/KMS BO Type

SCANOUT BUFFERS

Software Hardware

Wayland/Weston Components Khronos w/ Wayland Ext. OSS Components Hardware Specific

  • 1. Render w/

OpenGL/ES 2.Commit buffers w/ eglSwapBuffers()

  • 3. Import w/ eglCreateImageKHR()
  • 6. Set composed

buffers as KMS BOs.

  • 5. Compose w/

OpenGL/ES

  • 4. Register

destination frame buffers allocated with GBM.

slide-8
SLIDE 8

wayland protocol

8

Rendering and Composition: Full Screen or Sprite Rendering

Wayland Client OpenGL/ES + wayland_egl GPU driver for Wayland Weston / DRM Compositor OpenGL/ES + WL Ext. GBM DRM/KMS GPU GPU driver for Wayland Display

wl_egl_window Type wl_egl_window Type

RENDERING DISPLAYING

wl_buffer Type wl_buffer Type

FRAME BUFFERS

gbm_surface Type

CLIENT BUFFERS

wl_buffer Type

FULL SCREEN

wl_buffer Type DRM/KMS BO Type

SCANOUT BUFFERS

Software Hardware

Wayland/Weston Components Khronos w/ Wayland Ext. OSS Components Hardware Specific

  • 1. Render w/

OpenGL/ES 2.Commit buffers w/ eglSwapBuffers()

  • 3. Import w/ gbm_bo_import()
  • 4. Set composed

buffers as KMS BOs.

slide-9
SLIDE 9

PORTING WESTON TO R-CAR

9

slide-10
SLIDE 10
  • 1. OpenGL/ES for Wayland/Weston
  • 2. Zero Copy Mechanism for Native Buffer

“Typically, hardware enabling includes modesetting/display and EGL/GLES2. On top of that, Wayland needs a way to share buffers efficiently between processes.”

http://wayland.freedesktop.org/architecture.html

10

What Are Required?

slide-11
SLIDE 11
  • Must support the following Native Display Types for

eglGetDisplay():

– wl_display for clients – gbm handle for Weston

  • Must support the following EGL_EXTENSIONs:

– EGL_KHR_image_pixmap – EGL_WL_bind_wayland_display

  • Must support the following Native Pixmap Type for

eglCreateImageKHR():

– EGL_WAYLAND_BUFFER_WL

  • Must support the following Wayland extension APIs:

– eglBindWaylandDisplayWL – eglUnbindWaylandDisplayWL – eglQueryWaylandBufferWL

11

Wayland Requirements for OpenGL/ES

slide-12
SLIDE 12

WSEGL for Wayland

Weston for Renesas R-Car

12

Wayland Client Renesas OpenGL/ES libgbm w/ KMS Backend wayland-kms

Wayland/Weston Components New OSS Components Standard OSS Components

IPC API Call

Wayland Client Support Wayland Server Support Weston (Wayland Server) Wayland Server Stub EGL with Wayland Extension Wayland Client Stub libdrm & libkms

Proprietary

wayland-egl

slide-13
SLIDE 13
  • 1. A client creates a wl_surface on the

server.

  • 2. The client attach a wl_buffer to the

created surface.

  • 3. The client submit the wl_buffer to the

server.

  • 4. The server takes the wl_buffer and

compose to the screen. All of above should happen in zero-copy manner!

13

Wayland Composition Revisited

slide-14
SLIDE 14
  • An abstract data type that represents a

reference to a pixel buffer.

  • 2 open source implementations:

– wl_shm : wayland standard

  • Based on Linux shared memory. Not physically contiguous.

– wl_drm : Mesa standard

  • Based on DRI. Possibly physically contiguous.
  • Weston understands wl_shm only. Wl_drm is Mesa
  • specific. Thus, wl_drm is not handled by Weston,

but by Mesa internally.

14

What is wl_buffer by the way?

slide-15
SLIDE 15
  • Requirements

– End-to-end Buffer Zero Copy – Physically Contiguous Memory

  • wl_drm?

– Implementation is too Mesa dependent.

  • Need more generic implementation.

15

Which wl_buffer implementation to use?

slide-16
SLIDE 16
  • KMS BO buffer type.

– https://github.com/thayama/wayland-kms – Based on wl_drm in Mesa.

  • Imports DMABUF via PRIME, a dma-buf

interface layer in DRM.

– Originally, we used DRM Handle, but we now use DMABUF instead.

  • Can directly pass video output from V4L2.

16

wl_kms

slide-17
SLIDE 17

17

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

slide-18
SLIDE 18

18

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

Buffer Allocation. Client creates rendering surface with EGL API. WSEGL allocates memory with KMS BO, and make avaialable to GPU.

slide-19
SLIDE 19

19

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

Rendering to the allocated buffer. GPU gets all details needed about the buffer via WSEGL.

slide-20
SLIDE 20

20

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

When a client calls eglSwapBuffers(), WSEGL commits a buffer to the server via Wayland. The details of the buffer is DMABUF fd, a stride, a size, and a pixelf ormat.

slide-21
SLIDE 21

21

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

When the server receives the buffer, it imports with eglCreateImageKHR().

slide-22
SLIDE 22

22

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

WSEGL gets details of the buffer from wayland- kms, and asks to import the given DMABUF. The buffer is then made available to GPU.

slide-23
SLIDE 23

23

Buffer Zero Copying with wl_kms

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES gl-renderer compositor- drm KMS BO User Space Kernel Space Wayland GPU Driver KMS Driver OpenGL/ES Wayland GPU Driver

Gl-renderer can now refer the buffer passed by the client, and composes a final output.

slide-24
SLIDE 24

WHY LINUX MEDIA CONTROLLER RENDERER?

24

slide-25
SLIDE 25
  • Applications are heading towards more and

more GPU intensive.

  • People want to use GPU for more advanced

UI, rather than a simple window composition.

– On the other hand, some people want to do more complex composition using GPU. J

  • GPU Offloading is one way.

– https://archive.fosdem.org/2014/schedule/ event/wayland_gpu/ – But, still premature for real products.

25

Motivation

slide-26
SLIDE 26
  • Not many embedded SoC has

multiple GPUs.

  • However, they often have

sophisticated hardware for video signal processing that allow to do 2D blending.

  • Why not use them?

26

Simpler Approach?

slide-27
SLIDE 27

LINUX MEDIA CONTROLLER FRAMEWORK

27

slide-28
SLIDE 28
  • Just a Video4Linux2 device.
  • Make V4L2 media device parameters

and pipelines configurable from user space.

– http://linuxtv.org/downloads/presentations/ summit_jun_2010/20100206-fosdem.pdf

  • Important keywords: Entities, Pads, and

Links.

28

What is Media Controller?

slide-29
SLIDE 29

29

Media Controller

Entity Entity 1 1 Entity 2 1

Link Entity Source Pad Sink Pad

May have multiple sink pads

slide-30
SLIDE 30
  • A standard Linux API to configure

complicated media devices from user space.

  • On Renesas R-Car, VSP1, a device for

video signal processing, is exposed via Media Controller API in Linux.

  • Zero-copy could be easily achieved via
  • DMABUF. Ideal for our use case.

– Can use wl_kms!

30

Why Media Controller?

slide-31
SLIDE 31
  • Supported Features

– Scaling – Cropping – Pixel Format Conversion – 4 to 1 Blending

  • As pipelines in VSP1 are configurable,

best suits to Media Controller Framework!

31

Renesas R-Car VSP1

slide-32
SLIDE 32

V4L2 RENDERER FOR WESTON

32

slide-33
SLIDE 33

V4L2 Renderer Support in Weston

33

Wayland Client Wayland Client Stub Weston (Wayland Server) Wayland Server Stub OpenGL/ES v4l2-renderer device for VSPD compositor-drm v4l2-renderer shell libgbm/ libdrm compositor- core GPU Kernel Driver DRM/KMS Driver Qt, EFL, etc.

Weston Components Wayland Components Other Components IPC API Call

User Space Kernel Space VSP V4L2 Driver

New Components

New components added to Weston. Plus Minimum changes to the existing compositor- drm to host a v4l2- renderer.

slide-34
SLIDE 34
  • Kept as minimum as possible.
  • Changes are to load v4l2-renderer, and

pass output buffers to v4l2-renderer. Almost same as those of for pixman- renderer.

  • Minor changes on output buffer

allocations; DMABUF export and a read permission to mmap’d output buffer are added.

34

Changes to compositor-drm

slide-35
SLIDE 35
  • Media device agnostic layer.
  • Does everything needed to import wl_kms

and wl_shm buffer to V4L2 Media Controller arena, i.e. DMABUF.

  • Calculations required to figure out source

regions and destination regions are done in v4l2-renderer.

  • Anything that are not media device specific is

handled in v4l2-renderer.

35

v4l2-renderer

slide-36
SLIDE 36
  • Media device specific layer.
  • Does everything specific to the media devices.

– Media Controller Framework requires the background knowledge of the underlying media devices.

  • Does actual job to compose surfaces specified as

DMABUF from v4l2-renderer.

36

V4L2 renderer device

slide-37
SLIDE 37

API Descriptions init Initialize a v4l2 media controller device. create_output Create an output. No buffer passed yet. set_output_buffer Set an output buffer for the output. create_surface Create a surface. No buffer passed yet. attach_buffer Set a buffer for the surface. begin_compose Begin a new composition. finish_compose Finish the composition. draw_view Compose the surface. get_capabilities Get capabilities of the V4L2 Renderer Device.

37

V4L2 Renderer Device API

slide-38
SLIDE 38
  • V4L2 Renderer is not official yet.

– https://github.com/thayama/weston

38

Current Status

slide-39
SLIDE 39

CONCLUSIONS

39

slide-40
SLIDE 40
  • Yes and No.
  • Configuring parameters and links are not
  • free. Not really great if we need to

configure media device very often.

– DRI could be alternative.

  • On the other hand, use of the Linux

standard features including DMABUF is great for extensibility and flexibility.

40

Was Media Controller the Right Choice?

slide-41
SLIDE 41
  • Yes and No.
  • Other renderers help you implementing new

renderer.

  • Geometry was the most complex part of Weston.

– Global coordinate, Output local coordinate, and view coordinate.

  • Renderers are responsible for understanding these

coordinates and rendering views to the correct location.

– Pixman-renderer can give you some idea; how complicated it is.

41

Was implementing new renderer easy?

slide-42
SLIDE 42

42

Coordinate System in Weston

Output 0 Output 1

  • paque

region

View position is in Global Coordinate. Rendering needs to take care of

  • utput local

coordinate. Opaque region is specified in View Coordinate.

slide-43
SLIDE 43

APPENDIX

43

slide-44
SLIDE 44

Components used in Renesas OpenGL/ES for Wayland

Components Descriptions EGL with Wayland Extension A thin layer to support Wayland specific EGL APIs and a native buffer type for eglCreateImageKHR(). https://github.com/thayama/libegl wayland-kms A subclass of wl_buffer for to pass KMS BO. Defines a wl_kms wayland protocol, and server side codes. https://github.com/thayama/wayland-kms libgbm w/ KMS Backend GBM frontend extracted from Mesa and used in Weston with a KMS Backend support. https://github.com/thayama/libgbm WSEGL for Wayland A bridge component between GPU and Wayland.

44