Presenting 2D Web Content in XR TPAC 2018 Overview 2D Web Content - - PowerPoint PPT Presentation

Presenting 2D Web Content in XR

TPAC 2018

2D Web Content in 3D XR Rendering Context

Overview

• Background
• Potential implementation paths forward
• Open questions, concerns

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Background - 2D Content in XR

• Overlays for handheld AR. (DOM to Screen)
• Formatted text, control panels, 2D video players (DOM to World)

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Background - Compositors, Quad Layers

New platforms and hardware, new features to explore for rendering 2D content securely and performantly. Compositors

• Definition, purpose
• Predictive rendering

Quad Layer

• Definition, purpose
• How they work, what that means for rendering behavior e.g. occlusion, depth sensitivity,

transparency

• How they might help solve existing problems around exposing DOM content to XR context

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

User Agents and Compositor Layers: CSS3D

Rendering context and transform-style: preserve-3d;

• Elements can request to be in the same 3D context as their parents and sibling elements.
• Issues around ordering, depth buffer are effectively avoided by returning 3D objects to a 2D

rendering context when certain CSS attributes are defined on elements in the rendering context.

Instantiate new rendering context for element with will-change: transform; property

• Moves element to its own compositor layer under the hood for transform, animation
• ptimization, etc.
• Not in all browsers, limited to max layers underlying platform can support.

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Background - DOM to Texture

History of DOM to Texture

• Not a new idea.
• First proposed as a way to render 2D web views in WebGL.
• No way to implement while safely mitigating security concerns around

access to cross-origin content pixel data, input injection.

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Options we investigated

• 1. Do nothing
• 2. Solve for immersive mode on handheld devices only
• 3. Option 2 + a UA-managed workaround for immersive mode on headset

devices

• 4. Option 2 + quad layers.
• 5. Revisit feasibility of DOM to texture

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Option 2 - DOM to Screen

• Layout and rendering is required, but should elements be required to be in

the DOM tree hierarchy?

• Prior art - fullscreen API
• Could be implemented as a quad layer?

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Option 4 - DOM to World

DOM quad layer

• Input - static or interactive?
• Support cross-origin content?
• Layout and rendering is required, but should elements be in the DOM tree?
• Differences in underlying platform support for quad layers?
• Composition order
• Occlusion
• Input injection
• How would we extend WebXR to support multiple layers?

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Inputs

In order to interact with DOM content, UA will need to:

• Continue sending pose data as normal through WebXR APIs
• Compute raycast intersection for each input source against a quad layer and

generating existing DOM events to 2D window coordinates.

2D Web Content in 3D XR Rendering Context

2D Web Content in 3D XR Rendering Context

Cross-Origin DOM Content

Visual Representation

• No new security concerns with quad layer implementation, parent

document does not get access to nested document pixel data Inputs

• Security concerns delivering pose data to both document contexts.
• Pose data must continue to go to parent document for basic scene

rendering

• Delivering input injection results in two browsing contexts getting the

same input data.

2D Web Content in 3D XR Rendering Context

Differences in underlying platform support - Platform Capabilities

Platfor m Quad Layer Occlusion Depth Buffer ARKit N N N ARCore N N N Google VR SDK N* N N Magic Leap N* N N Oculus Y, multiple (ovrLayerQ uad) Y Y Platform Quad Layer Occlusion Depth Buffer OpenVR Y, limited (IVRComposit

• r::Submit)

N* N OpenXR Y, multiple (XrCompositor LayerQuad) Y* Y* WMR Hololens N (MaxQuadLay ers set to 0) N N WMR Immersive Headsets Y, multiple (HolographicQ uadLayer) Only with

• ther quad

layers N*

2D Web Content in 3D XR Rendering Context

Next Steps

• Rationalize a common abstraction for a quad layer API across different

platforms

• What changes are necessary for the WebXR API to support multiple layers
• If we add support for a new XR layer type, is this the only time it will be

necessary to do so or are there other features that will need their own layer abstraction?

• Is a quad layer abstraction sufficient for both DOM to screen and DOM to

world?

• Identifying collaborators for design, action items

2D Web Content in 3D XR Rendering Context