Kernel Foveated Rendering Xiaoxu Meng, Ruofei Du, Matthias Zwicker - PowerPoint PPT Presentation

Kernel Foveated Rendering Xiaoxu Meng, Ruofei Du, Matthias Zwicker and Amitabh Varshney Augmentarium | UMIACS 1 University of Maryland, College Park

Introduction Our Approach User Study Experiments Conclusion Related Work Resolution Frame rate MPixels / sec Application 1920 x 1080 x 1 60 124 Desktop game 2

Introduction Our Approach User Study Experiments Conclusion Related Work Resolution Frame rate MPixels / sec Application 1920 x 1080 x 1 60 124 Desktop game 2018 VR 1440 x 1600 x 2 90 414 (HTC Vive PRO) 3

Introduction Our Approach User Study Experiments Conclusion Related Work Resolution Frame rate MPixels / sec Application 1920 x 1080 x 1 60 124 Desktop game 2018 VR 1440 x 1600 x 2 90 414 (HTC Vive PRO) 4000 x 4000 x 2 90 2,880 2020 VR * 4 * Data from Siggraph Asia 2016, Prediction by Michael Abrash, October 2016

Introduction Our Approach User Study Experiments Conclusion Related Work 2000 1800 1600 • Virtual reality is a challenging workload 1400 Mpixels/sec 1200 1000 800 600 400 200 0 Desktop Game 2017 VR 2020 VR 5

Introduction Our Approach User Study Experiments Conclusion Related Work • Virtual reality is a challenging workload • Most VR pixels are peripheral fovea: the center of the retina corresponds to the center of the vision field 6

Introduction Our Approach User Study Experiments Conclusion Related Work • Virtual reality is a challenging workload • Most VR pixels are peripheral foveal region: the human eye detects significant detail peripheral region: the human eye detects little high fidelity detail 7

Introduction Our Approach User Study Experiments Conclusion Related Work • Virtual reality is a challenging workload • Most VR pixels are peripheral foveal foveal region region foveal region: the human eye detects significant detail peripheral region: the human eye detects little high fidelity detail 8

Introduction Our Approach User Study Experiments Conclusion Related Work Percentage of the foveal pixels 96 % 100% 90% • Virtual reality is a challenging workload 80% 70% 60% • Most VR pixels are peripheral 50% 40% 27 % 30% 4 % 20% 10% 0% iPhone7 Plus 27'' Desktop Monitor 2016 VR HMD 9 * Data from Siggraph 2017, by Anjul Patney, August 2017

10

Foveated Rendering 11

Introduction Our Approach User Study Experiments Conclusion Related Work • Virtual reality is a challenging workload • Most VR pixels are peripheral Eye tracking technology available • 12

Related Work 13

Introduction Our Approach User Study Experiments Conclusion Related Work Multi-Pass Foveated Rendering [Guenter et al. 2012] 𝟐 𝟐 𝟓 Resolution Full Resolution 𝟑 Resolution 14

Introduction Our Approach User Study Experiments Conclusion Related Work Coarse Pixel Shading (CPS) [Vaidyanathan et al. 2014] Input primitives Rasterizer Early Z Evaluate Coarse 𝑈 𝑦 × 𝑈 𝑧 Pixel Size Tile Buffer Generate Coarse Quad Shade 15

Introduction Our Approach User Study Experiments Conclusion Related Work CPS with TAA & Contrast Preservation [Patney et al. 2016] 16

Introduction Our Approach User Study Experiments Conclusion Related Work Can we change the resolution gradually? 17

Introduction Our Approach User Study Experiments Conclusion Related Work Perceptual Foveated Rendering [Stengel et al. 2016] 18

Introduction Our Approach User Study Experiments Conclusion Related Work Is there a foveated rendering approach without the expensive pixel interpolation? 19

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] 𝑧 𝑤 2𝜌 (𝑦 0 , 𝑧 0 ) 𝑣 𝑤 𝑃 𝑦 (𝑦 0 , 𝑧 0 ) 𝑣 𝑀 Cartesian coordinates Log-polar coordinates (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 ∙ 𝑥 𝑀 (arctan 𝑧 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 20 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates (𝑦, 𝑧) (u, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 ∙ 𝑥 𝑀 (arctan 𝑧 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 21 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 ∙ 𝑥 𝑀 (arctan 𝑧 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 22 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates Cartesian coordinates (𝑦, 𝑧) (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 𝑦 = 𝑓 𝑀∙𝑣 𝑥 cos 𝑤 ∙ 2𝜌 ∙ 𝑥 ℎ 𝑀 (arctan 𝑧 𝑧 = 𝑓 𝑀∙𝑣 𝑥 sin 𝑤 ∙ 2𝜌 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 23 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates Cartesian coordinates (𝑦, 𝑧) (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 𝑦 = 𝑓 𝑀∙𝑣 𝑥 cos 𝑤 ∙ 2𝜌 ∙ 𝑥 ℎ 𝑀 (arctan 𝑧 𝑧 = 𝑓 𝑀∙𝑣 𝑥 sin 𝑤 ∙ 2𝜌 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 24 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates Cartesian coordinates (𝑦, 𝑧) (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 𝑦 = 𝑓 𝑀∙𝑣 𝑥 cos 𝑤 ∙ 2𝜌 ∙ 𝑥 ℎ 𝑀 (arctan 𝑧 𝑧 = 𝑓 𝑀∙𝑣 𝑥 sin 𝑤 ∙ 2𝜌 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 25 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar mapping [Araujo and Dias 1996] Cartesian coordinates Log-polar coordinates Cartesian coordinates (𝑦, 𝑧) (𝑦, 𝑧) (𝑣, 𝑤) Log-polar Mapping 𝑣 = log 𝑦 2 + 𝑧 2 𝑦 = 𝑓 𝑀∙𝑣 𝑥 cos 𝑤 ∙ 2𝜌 ∙ 𝑥 ℎ 𝑀 (arctan 𝑧 𝑧 = 𝑓 𝑀∙𝑣 𝑥 sin 𝑤 ∙ 2𝜌 𝑦 + 𝟐 [𝑧 − 0] ∙ 2𝜌) 𝑤 = ∙ ℎ ℎ 2𝜌 • 𝑋: 𝑡𝑑𝑠𝑓𝑓𝑜 𝑥𝑗𝑒𝑢ℎ 𝐼: 𝑡𝑑𝑠𝑓𝑓𝑜 ℎ𝑓𝑗𝑕ℎ𝑢 𝑥: 𝑐𝑣𝑔𝑔𝑓𝑠 𝑥𝑗𝑒𝑢ℎ ℎ: 𝑐𝑣𝑔𝑔𝑓𝑠 ℎ𝑓𝑗𝑕ℎ𝑢 𝟐 𝑧 < 0 = ቊ1 𝑧 < 0 • 0 𝑧 > 0 26 𝑀 = log 𝑋 2 + 𝐼 2 •

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar Mapping for 2D Image [Antonelli et al. 2015] 27

Introduction Our Approach User Study Experiments Conclusion Related Work Log-polar Mapping for 2D Image 28

Introduction Our Approach User Study Experiments Conclusion Related Work Our Approach 29