Tell Your Graphics Stack That the Display Is Circular Hongyu Miao - - PowerPoint PPT Presentation

▶
tell your graphics stack that
SMART_READER_LITE
LIVE PREVIEW

Tell Your Graphics Stack That the Display Is Circular Hongyu Miao - - PowerPoint PPT Presentation

Sep 28, 2023 232 likes •802 views

Tell Your Graphics Stack That the Display Is Circular Hongyu Miao Felix Xiaozhu Lin miaoh@purdue.edu xzl@purdue.edu (1975) Text Display Rectangular 2 (1980s) RGB Color Display Rectangular 3 Modern Desktop: Rectangular

slide-1
SLIDE 1

Tell Your Graphics Stack That the Display Is Circular

Hongyu Miao Felix Xiaozhu Lin miaoh@purdue.edu xzl@purdue.edu

slide-2
SLIDE 2

(1975) Text Display Rectangular

2

slide-3
SLIDE 3

(1980s) RGB Color Display Rectangular

3

slide-4
SLIDE 4

Modern Desktop: Rectangular

4

slide-5
SLIDE 5

Smartphone Display: Rectangular

5

slide-6
SLIDE 6

(2016) Wearable Displays Circular!

6

slide-7
SLIDE 7

Display hardware: Not only rectangular any longer!

Software Challenges

7

slide-8
SLIDE 8

Graphics Stack: Rectangular area

What’s the problem?

8

slide-9
SLIDE 9

Graphics Stack: Rectangular area

What’s the problem?

9

slide-10
SLIDE 10

Displayed: Circular area

What’s the problem?

Graphics Stack: Rectangular area

10

slide-11
SLIDE 11

Invisible area: WASTED!

What’s the problem?

Displayed: Circular area Graphics Stack: Rectangular area

11

slide-12
SLIDE 12

A 20,000 foot view of graphics stack

Shader

12

slide-13
SLIDE 13

Shader

A 20,000 foot view of graphics stack

13

PhoneWindowsDe corView LinearLayout FrameLayout FrameLayout TextView Button TextView

slide-14
SLIDE 14

Shader

A 20,000 foot view of graphics stack

Save DrawRenderNode DrawRenderNode DrawRect Display List

14

slide-15
SLIDE 15

Shader

A 20,000 foot view of graphics stack

Save DrawRenderNode DrawRenderNode DrawRect Display List

glDisable(cap=GL_SCISSOR_TEST) glActiveTexture(texture=GL_TEX TURE0) glGenBuffer(n=1,buffer=[2]) glBindBuffer(target=GL_ARRAY_B UFFER,buffer=2) OpenGL Commands

15

slide-16
SLIDE 16

Shader

A 20,000 foot view of graphics stack

16

  • 1. Load texture
slide-17
SLIDE 17

Shader

A 20,000 foot view of graphics stack

17

. . . .

  • 1. Load texture
  • 2. Read vertex data

2

slide-18
SLIDE 18

Shader

A 20,000 foot view of graphics stack

18

. . . .

  • 1. Load texture
  • 2. Read vertex data
  • 3. Execute vertex shader

2,3

slide-19
SLIDE 19

Shader

A 20,000 foot view of graphics stack

19

. . . . . . . .

  • 1. Load texture
  • 2. Read vertex data
  • 3. Execute vertex shader
  • 4. Assemble vertices

2,3 4

slide-20
SLIDE 20

Shader

A 20,000 foot view of graphics stack

20

. . . . . . . . . . . .

  • 1. Load texture
  • 2. Read vertex data
  • 3. Execute vertex shader
  • 4. Assemble vertices
  • 5. Rasterize into fragments

2,3 4 5

slide-21
SLIDE 21

Shader

A 20,000 foot view of graphics stack

21

. . . . . . . . . . . . . . . .

  • 1. Load texture
  • 2. Read vertex data
  • 3. Execute vertex shader
  • 4. Assemble vertices
  • 5. Rasterize into fragments
  • 6. Execute fragment shader

2,3 4 5 6

slide-22
SLIDE 22

Shader

A 20,000 foot view of graphics stack

22

. . . . . . . . . . . . . . . . . . . .

  • 1. Load texture
  • 2. Read vertex data
  • 3. Execute vertex shader
  • 4. Assemble vertices
  • 5. Rasterize into fragments
  • 6. Execute fragment shader
  • 7. Write to frame buffer

2,3 4 5 6 6,7

slide-23
SLIDE 23

Shader

A 20,000 foot view of graphics stack

Save DrawRenderNode DrawRenderNode DrawRect Display List

glDisable(cap=GL_SCISSOR_TEST) glActiveTexture(texture=GL_TEXTURE0) glGenBuffer(n=1,buffer=[2]) glBindBuffer(target=GL_ARRAY_BUFFER,buffer=2) OpenGl Commands

23

slide-24
SLIDE 24

Graphics stack is oblivious to display shape

app evidence

(0, 0) (MaxX, 0) (0, MaxY) (MaxX, MaxY)

24

slide-25
SLIDE 25

Texture is specified as a rectangular

Graphics stack is oblivious to display shape

OpenGL evidence

25

slide-26
SLIDE 26

Graphics stack is oblivious to display shape

Device driver evidence

Device tree code from Linux kernel (for LG Watch R)

26

slide-27
SLIDE 27

Top questions

How many resources are wasted? How should existing graphics stack adapt?

27

slide-28
SLIDE 28

Top questions

How many resources are wasted? How should existing graphics stack adapt?

28

slide-29
SLIDE 29

UI elements hidden & clipped by display boundary

29

slide-30
SLIDE 30

Wasted CPU & GPU computation

Drawing Upload texture: 25ms Shader compile:8.2ms Shader link: 1.2ms Other: 2.4ms Rendering time: 4.5ms

31

slide-31
SLIDE 31

Wasted memory traffic

32

slide-32
SLIDE 32

Wasted memory traffic

33

slide-33
SLIDE 33

Wasted memory traffic

34

slide-34
SLIDE 34

Top questions

How many resources are wasted?

  • Few views are completely hidden
  • Not too much GPU/CPU computation is wasted
  • Much memory traffic is wasted

35

slide-35
SLIDE 35

Study of tens of wearable Apps

Apps # of UI Views Drawing Time

Rdr. Time

Hidden Clipped Total Shader compile Shader link† Texture upload† Other cmds Google keep

   8.6 1.3 4.4 2.9 4.3

Attopedia

9 10 8.2 1.2 25.0 2.4 4.5

Hole19

5 8 30.4 1.1 4.9 4.1 2.6

WearbottleSpinner

4 5 18.0 3.2 116.2 2.1 3.0

GridViewPager

6 9 23.9 4.4 2.0 2.0 2.8

Runtastic*

14 17

  • 3.9

ReminderByTime*

13 14

  • 3.8

Fit*

13 16

  • 3.3

Weatherlive*

14 17

  • 4.6

Instaweather*

13 16

  • 3.8

Hangout*

13 16

  • 3.7

36

slide-36
SLIDE 36

Top questions

How many resources are wasted? How should existing graphics stack adapt?

37

slide-37
SLIDE 37

Key: which layer should be aware

  • f display shape

38

slide-38
SLIDE 38

Developer-controlled layout Tedious & not portable Complicated UI library Tens of thousands of SLoC to be changed

Key: which layer should be aware

  • f display shape

39

slide-39
SLIDE 39

Developer-controlled layout Tedious & not portable Complicated UI library Tens of thousands of SLoC to be changed Cannot reduces wastes above this layer LPD [ATC15] Incomplete

Key: which layer should be aware

  • f display shape

40

slide-40
SLIDE 40

Pilot solution: OpenGL interposition

  • Key point: rewrite shader program one-the-fly

(x0,y0) (x,y) r

void main() { if((x-x0)*(x-x0)+(y-y0)*(y-y0)<r*r){ //Render the pixel if in circular area gl_FragColor = \ texture2D(textureUnit, textureCoordinate); ... ... } }

Shader

41

slide-41
SLIDE 41

Pilot solution: OpenGL interposition

Before Rewriting Shader After Rewriting Shader

42

slide-42
SLIDE 42

Evaluation: setup (ideal)

Profiling data stream Benchmark app Mobile GPU Profiler

Desktop Circular Watch

43

slide-43
SLIDE 43

Qualcomm’s GPU Profiler for Adreno

44

slide-44
SLIDE 44

Evaluation: setup (actual)

Profiling data stream Mobile GPU Profiler

Desktop Nexus 5

Similar QCOM SoC Same-generation GPU

46

slide-45
SLIDE 45

Result: Reduced GPU memory read

Lower is better!

47

slide-46
SLIDE 46

Result: Reduced GPU cycles

Lower is better!

48

slide-47
SLIDE 47

Estimated Power Saving

LPD [ATC15] Reduced DRAM-to-display traffic 7 MBps Saved power 2.7 mW Our method Reduced memory traffic 10 MBps Will save power 3.9 mW Our method (if novel display controller) Will reduce memory traffic 15 MBps Will save power 5.8 mW Assumption: Power saving roughly proportional to reduced traffic

49

slide-48
SLIDE 48

Summary

How many resources are wasted? How should existing graphics stack adapt?

  • Graphics stack is wasting resources due to screen shape
  • Quantified the resource wasted on the LG watch R
  • Pilot solution: interposing OpenGL + shader program
  • Reduced 22.4% memory traffic + 11.8% GPU cycles

50

slide-49
SLIDE 49

Outlook: future irregular displays

51

slide-50
SLIDE 50

Dashboard

Outlook: future irregular displays

52

slide-51
SLIDE 51

Virtual Reality Helmet

Outlook: future irregular displays

53

slide-52
SLIDE 52

Designing for future irregular displays

  • Higher waste → compelling to adapt graphics stack
  • Redesigning graphics stack may be justified

A key lesson

  • New form factors drive system software design

54

slide-53
SLIDE 53

Summary

How many resources are wasted? How should existing graphics stack adapt?

  • Graphics stack is wasting resources due to screen shape
  • Quantified the resource wasted on the LG watch R
  • Pilot solution: interposing OpenGL + shader program
  • Reduced 22.4% memory traffic + 11.8% GPU cycles

Future thoughts

  • New form factors drive system software design

55

slide-54
SLIDE 54

Q/A

  • How often does the waste occur?
  • Why can’t developers just design for irregular displays?
  • Why should we care about 10mW?
  • Why can’t you measure the power physically?
  • Can’t we just overhaul the UI library?
  • (From one Microsoft guy)
  • Can you solve this problem completely?
  • How do you rewrite the GPU shaders?

56

slide-55
SLIDE 55