[PPT] - FOSS Virtual & Augmented Reality Monado & OpenXR Jakob PowerPoint Presentation

SLIDE 1

1

FOSS Virtual & Augmented Reality

Monado & OpenXR

Jakob Bornecrantz jakob@collabora.com 01/02/2020

SLIDE 2

2

Open First

Jakob Bornecrantz

FDO GH @Wallbraker jakob@collabora.com

SLIDE 3

3

What is AR & VR
OpenXR
Monado
Status
Demos
Wrapping up

SLIDE 4

4

What is AR & VR

OpenXR Monado Status Demos Wrapping up

SLIDE 5

5

What is Augmented Reality?

Augment

━ Verb ━ /ɔːɡˈmɛnt/ ━ make greater by adding to it; increase

SLIDE 6

6

Open First

SLIDE 7

7

Open First

SLIDE 8

8

What is Virtual Reality?

I reject your reality and substitute my own

━ Dungeonmaster, 1984

■ (popularized by Adam Savage)

SLIDE 9

9

Open First

SLIDE 10

10

Open First

SLIDE 11

11

AR + VR = ?

Also known as XR
X = A | V
Not eXtended Reality

SLIDE 12

12

What is AR & VR

OpenXR

Monado Status Demos Wrapping up

SLIDE 13

13

Program Platform Hardware

How do we XR?

SLIDE 14

14

Program Platform Hardware

How do we XR? API

SLIDE 15

15

SLIDE 16

16

It’s just like OpenGL & Vulkan

━ Made by Khronos

Out now
Action Based
FOSS Implementation! :D

What’s cool about OpenXR?

SLIDE 17

17

Type of controller not in focus

━ Only after the fact

Context

━ From program ━ To runtime ━ Runtime decides what button does what

Action Based

SLIDE 18

18

What is AR & VR OpenXR

Monado

Status Demos Wrapping up

SLIDE 19

19

Program Platform Hardware

How do we XR?

SLIDE 20

20

Program Monado Hardware

Where does Monado fit in?

SLIDE 21

21

OpenXR State Tracker Drivers Compositor Program

xrt_compositor xrt_device OpenXR

Platform

xrt_prober

Monado

Hardware

SLIDE 22

22

OpenXR State Tracker Drivers

GL Wrapper

Program

xrt_gl_compositor xrt_device OpenXR

Platform

xrt_prober

Complex Truth

OpenXR Loader

Compositor

xrt_fd_compositor

Hardware Auxiliary

SLIDE 23

23

OpenXR State Tracker Hardware Compositor Drivers

xrt_compositor xrt_device Vulkan, X11, Wayland and KMS

Platform

xrt_prober

Linux Hardware

Linux APIs Misc APIs

Program

OpenXR

Desktop Stack

SLIDE 24

24

OpenXR State Tracker IPC Layer Hardware Compositor Drivers

xrt_fd_compositor xrt_device xrt_fd_compositor xrt_device Vulkan and KMS

Platform

xrt_prober xrt_prober

Linux Hardware

Linux APIs Misc APIs Process boundary

With IPC - Anticipated

SLIDE 25

25

Between different components

━ Auxiliary code has their own interfaces

Purely C
Not stable![1]

XRT Interfaces

SLIDE 26

26

HMDs
Controllers
Aspect pattern

━ Optional structs ━ HMD info

xrt_device

Functions

━ Input Sync ━ Output Set ━ View/Position Retrieval

Info (immutable)

━ Input/Outputs ━ Tracking Origin ━ Screen position (HMD) ━ Distortion (HMD)

SLIDE 27

27

Base class

━ xrt_[gl|vk|fd]_compositor ━ GL & VK are wrappers

■ Client side only

━ FD is “real” compositor

■ Written in Vulkan ■ Only “talks” in FDs

xrt_swapchain

━ xrt_[gl|vk|fd]_swapchain ━ acquire, wait & release

xrt_compositor

Functions

━ More explicit then OpenXR ━ Create Swapchain ━ Prepare, Begin & End ━ Wait, Begin, Discard & End ━ Poll Events (Refactor?)

Info (immutable)

━ Supported Formats

SLIDE 28

28

Policy holder

━ Reads configuration ━ Decides what devices open

Device prober

━ Used by OpenXR ST

Platform code

━ Abstract native interfaces ━ Used by device drivers

Should be split in two

━ xrt_prober ━ xrt_platform

xrt_prober

Functions

━ Look for devices ━ Select (opens devices) ━ List & Open Video devices ━ Misc USB

Info (immutable)

━ None

SLIDE 29

29

What is AR & VR OpenXR Monado

Status

Demos Wrapping up

SLIDE 30

30

Mostly complete OpenXR support
In-process Compositor

━ Mesh shader

Video processing framework
Tracking PS Move
Nifty debug UI

Status

SLIDE 31

31

North Star*
Daydream Ctrl*
OSVR HDK
Vive (Pro) & Index

━ Positional tracking through libsurvive

PlayStation VR
PlayStation Move
Razer Hydra
OpenHMD

━ Port to native drivers where it makes sense.

Hardware support

SLIDE 32

32

35.5K lines of codes

━ Drivers: 6.9K

■

9 drivers

■

Vive: 1.5K

■

HDK: 0.5K

━ Compositor: 6.6K ━ Auxiliary: 7.6K ━ State Trackers: 10.1K ━ Targets: 1.3K

Code

SLIDE 33

33

Complete OpenXR support
Improve PS Move Tracking
PSVR Tracking
Out of process compositor

Next Up

SLIDE 34

34

Safe space
System UI
Setup UI
Lighthouse Tracking
AR and SLAM

Even further out

SLIDE 35

35

What is AR & VR OpenXR Monado Status

Demos

Wrapping up

SLIDE 36

36

Tracking

SLIDE 37

37

Debug UI

SLIDE 38

38

Tracking (back-up)

SLIDE 39

39

What is AR & VR OpenXR Monado Status Demos

Wrapping up

SLIDE 40

40

Gnome-Shell

━ Output hot-plug ━ non-desktop

Intel Missing Extension

━ GL_EXT_memory_object_fd

Rant

SLIDE 41

41

Lots of work needed
GNOME & KDE UIs?

━ Monado interface library?

Switching between apps?

Some thoughts

SLIDE 42

42

FOSS XR Community @FossXR

SLIDE 43

43

Monado

━ https://monado.dev/ ━ https://monado.pages.freedesktop.org/monado ━ https://gitlab.freedesktop.org/monado/monado

OpenXR

━ https://www.khronos.org/openxr

Work on GFX & XR

━ Internships @ Collabora ━ Work @ Collabora

Links

SLIDE 44

44

Open First

Talk to me about

XR, VR & AR
FOSS XR
Volt Programming Language
FPGAs & FOSS
Amiga (FPGA), mc68k (LLVM)
Voxel/SVO rendering
FOSS & Society
Joining Collabora!

SLIDE 45

45

FOSS Virtual & Augmented Reality

Monado & OpenXR

Jakob Bornecrantz

FDO GH @Wallbraker jakob@collabora.com

OpenXR Monado Status Demos Wrapping up

What is Augmented Reality?

━ Verb ━ /ɔːɡˈmɛnt/ ━ make greater by adding to it; increase

What is Virtual Reality?

━ Dungeonmaster, 1984

AR + VR = ?

What is AR & VR

Monado Status Demos Wrapping up

Program Platform Hardware

How do we XR?

Program Platform Hardware

How do we XR? API

━ Made by Khronos

What’s cool about OpenXR?

━ Only after the fact

━ From program ━ To runtime ━ Runtime decides what button does what

Action Based

What is AR & VR OpenXR

Status Demos Wrapping up

Program Platform Hardware

How do we XR?

Program Monado Hardware

Where does Monado fit in?

OpenXR State Tracker Drivers Compositor Program

Platform

Monado

Hardware

OpenXR State Tracker Drivers

Program

Platform

Complex Truth

Hardware Auxiliary

Program

Desktop Stack

With IPC - Anticipated

━ Auxiliary code has their own interfaces

XRT Interfaces

xrt_device

xrt_compositor

xrt_prober

What is AR & VR OpenXR Monado

Demos Wrapping up

━ Mesh shader

Status

━ Positional tracking through libsurvive

━ Port to native drivers where it makes sense.

Hardware support

━ Drivers: 6.9K

━ Compositor: 6.6K ━ Auxiliary: 7.6K ━ State Trackers: 10.1K ━ Targets: 1.3K

Code

Next Up

Even further out

What is AR & VR OpenXR Monado Status

Wrapping up

Tracking

Debug UI

Tracking (back-up)

What is AR & VR OpenXR Monado Status Demos

━ Output hot-plug ━ non-desktop

━ GL_EXT_memory_object_fd

Rant

━ Monado interface library?

Some thoughts

FOSS XR Community @FossXR

━ https://monado.dev/ ━ https://monado.pages.freedesktop.org/monado ━ https://gitlab.freedesktop.org/monado/monado

━ https://www.khronos.org/openxr

━ Internships @ Collabora ━ Work @ Collabora

Links

Talk to me about

FOSS Virtual & Augmented Reality

Questions?