Greenfield The in-browser Wayland compositor Introduction $ - PowerPoint PPT Presentation

Greenfield The in-browser Wayland compositor

Introduction ● $ whoami ○ Erik De Rijcke ○ Self employed ; udev.be ○ [ Kotlin, Java ] @ Day ; [ JavaScript, C, … ] @ Night $ which greenfield ● ○ Wayland compositor JavaScript 😲 ○ ■ ES6 File per class ■ ■ JSDoc type comments WebAssembly ○ ■ Native libraries available in the browser

What does Greenfield offer? != == ● A resumable screen forwarding solution à ● Something super awesome! la VNC/RDP/Citrix ● Pure JavaScript and a wee bit of ● Something finished WebAssembly. No plugins required, all HTML5. ● A per application remote rendering solution - but not limited to that... A true cloud ☁ desktop environment ● ● A work in progress.

First some basic Compositor concepts explained - A Wayland compositor Wayland App. - binary wire protocol - also FDs - WebRTC - p2p communication Signaling - needs central server signaling/broker Peer A Peer B Peer Connection

How does 154.19.251.2 Greenfield work? Browser Compositor Browser ● ○ Wayland compositor C o Westfield ● m ○ Underlying library p JavaScript protocol generator ○ o ● Application Endpoint s i t Node.js ⇒ Rust? ○ o r ○ Stateless proxy compositor Libwayland-server fork ○ 2001:db8:0:1234:0:567:8:1 172.16.254.1 ○ Forwards to browser or to App. App. native libraries Endpoint. Endpoint. ○ Encodes application frames Connection per wayland application ● ○ WebRTC data channel (WebSocket) ○ Wayland Wayland ● Encoded application content App. App. h264 ○ ○ jpeg

Wayland Buffer with alpha channel Life of an h264 GStreamer application frame Encoded image YUV Encoded image A Network ● Need precise encoding/decoding semantics Encoded image YUV Encoded image A ○ Video streaming Commit solutions unsuited! WASM h264 decoder ● Encoding frame ○ Gstreamer Decoded image YUV Decoded image A ● Decoding ○ WebAssembly h264 WebGL Alpha & Color Conversion software decoder ● Frame rendering is throttled ○ long pipeline HTML5 ImageBitmap ○ full round-trip takes a lot of time HTML5 Canvas (back-buffer) ● Possibility to “parallelize” pipeline requestAnimationFrame ○ hard to predict speed of frame in pipeline HTML5 Canvas (front-buffer)

Running 154.19.251.2 applications in a JS or JS web worker* Browser Wasm Client Compositor Wayland library App. Web worker runs a JavaScript ● C o or WebAssemly Wayland app. m No network latency ○ p ○ Zero copy data transfer o s All the good stuff! ○ i t ● Offscreen WebGL o r Needs special protocol, ○ much like wl_drm 2001:db8:0:1234:0:567:8:1 172.16.254.1 App. toolkit ○ App. App. ■ Skia WASM Endpoint. Endpoint. Apps. to WASM ○ ● Cloud desktop environment Account based ○ application access Wayland Wayland WebApp. store ○ App. App. ○ WebApp. repositories *work in progress

Demo Time!

KThxBye! Contact: You have Erik De Rijcke: derijcke.erik@gmail.com questions! udev.be BVBA: www.udev.be info@udev.be