for Ubiquitous Multi-device Interaction Sangeun Oh , Ahyeon Kim , - - PowerPoint PPT Presentation

for ubiquitous multi device interaction
SMART_READER_LITE
LIVE PREVIEW

for Ubiquitous Multi-device Interaction Sangeun Oh , Ahyeon Kim , - - PowerPoint PPT Presentation

FLUID: Flexible User Interface Distribution for Ubiquitous Multi-device Interaction Sangeun Oh , Ahyeon Kim , Sunjae Lee , Kilho Lee , Dae R. Jeong , Steven Y. Ko , and Insik Shin Various surfaces become


slide-1
SLIDE 1

FLUID: Flexible User Interface Distribution for Ubiquitous Multi-device Interaction

Sangeun Oh ⃰, Ahyeon Kim ⃰, Sunjae Lee ⃰, Kilho Lee ⃰, Dae R. Jeong ⃰, Steven Y. Ko†, and Insik Shin ⃰

slide-2
SLIDE 2

Various surfaces become pervasive!

  • Smart devices have various surfaces with different shapes & sizes

– From smartwatch to smart TV – Foldable screen (Samsung Galaxy Fold) / dual screen (LG V50)

2

slide-3
SLIDE 3

Potential for multi-surface interaction

  • The trend can change how users interact with applications

– Using only single surface

3

User Single surface Application

 Using multiple surfaces concurrently

slide-4
SLIDE 4

Multiple surfaces

Potential for multi-surface interaction

  • The trend can change how users interact with applications

– Using only single surface

4

User Application

 Using multiple surfaces concurrently

slide-5
SLIDE 5

Use case: live streaming

5

slide-6
SLIDE 6

Existing solutions

  • Customized apps

– Extra engineering efforts – Low applicability

  • Screen mirroring

– Low flexibility

  • Supports only full screen

– Low responsiveness for high resolutions

  • App migration

– Low flexibility

  • Supports only full screen
  • Cannot support concurrent usage

6

Flux [EuroSys’15] Vysor Chromecast Netflix Smartwatch apps Google docs

slide-7
SLIDE 7

Research goal

  • Design a new mobile platform that supports multi-surface interaction

by distributing UI objects to different devices

– in a flexible, transparent and responsive manner

7

FLUID (FLexible UI Distribution)

slide-8
SLIDE 8

UI object A

Target App

UI object B

<Host device>

App logic

Surface

FLUID overview

  • Key idea: separation between app logic & UI parts

8

UI update Rendering Input event User

slide-9
SLIDE 9

UI selection Target App

UI object B

<Host device>

App logic

Surface

FLUID overview

  • Key idea: separation between app logic & UI parts

1) Distributing target UI objects to remote devices and rendering them

9

FLUID Wrapper App <Guest device> Surface Rendering

UI object A UI object A

slide-10
SLIDE 10

Target App

UI object B

<Host device>

App logic

Surface

FLUID overview

  • Key idea: separation between app logic & UI parts

1) Distributing target UI objects to remote devices and rendering them

10

FLUID Wrapper App <Guest device> Surface Rendering Rendering UI distribution

UI object A UI object A

slide-11
SLIDE 11

Target App <Host device> Surface

FLUID overview

  • Key idea: separation between app logic & UI parts

1) Distributing target UI objects to remote devices and rendering them 2) Giving an illusion as if app logic and UI objects were in the same process

11

FLUID Wrapper App <Guest device> Surface

UI object B App logic UI object A

Rendering Rendering Input event User UI update

Same-process illusion

slide-12
SLIDE 12

Why is FLUID good?

  • Flexibility

– Allow users to control multiple surfaces as they want via fine-grained UI distribution

  • Transparency

– Support legacy apps without any modification to them – Develop new multi-surface apps under the existing programming model

  • Responsiveness

– Require less network transmission when moving UIs instead of full screen

12

slide-13
SLIDE 13

Problems

  • P1. How to split & distribute UI objects?

– Transmits minimum graphical states needed for UI rendering

  • To reduce network overhead

– However, it is unknown which graphical states app-specific custom UIs use

13

Android UI class Custom UI class Unknown member fields

?

slide-14
SLIDE 14

Problems

  • P1. How to split & distribute UI objects?

– Transmits minimum graphical states needed for UI rendering

  • To reduce network overhead

– However, it is unknown which graphical states app-specific custom UIs use – Our solution: Selective UI distribution

14

FLUID static analyzer FLUID platform

Platform image Target app’s APK file Analysis results

UI object B UI object A

Runtime tracking <Offline stage> <Online stage>

slide-15
SLIDE 15

Problems

  • P2. How to maintain interaction between app logic & UI objects?

– Such interaction is achieved via local function calls

  • e.g., TextView.setText(), ImageView.setImageResource(), etc.

– However, local functions cannot be executed across devices – Our solution: transparent RPC transformation in Android VM (ART)

15

<Host device>

App logic UI object A UI object A

Migrated <Guest device> Local function call Remote Procedure Call (RPC) Call intercept!

slide-16
SLIDE 16

Evaluation environment

  • Implemented FLUID prototype based on Android 8.1 (Oreo)
  • Used Google Pixel XL (smartphone) & Pixel C (tablet)

– Phone-to-phone – Phone-to-tablet – Tablet-to-phone

  • On the Same WiFi network

16

Pixel XL Pixel C

slide-17
SLIDE 17

App coverage

  • Using 20 legacy apps for 10 multi-

surface scenarios

– All legacy apps use their own custom UIs

  • FLUID can support various legacy

apps successfully

17

Use case scenario UI type App name Login with personal device

Editor Instagram Paypal

Fill in information collaboratively

Text, editor eBay Booking.com

Chatting with different device while broadcasting

Button, editor LiveMe Afreeca TV

Search destination with different device

Button, editor Naver map Maps.me

Control media with different device

Seek bar, button VLC Player Music Player

Control painting tool with different device

Scroll, image, button PaperDraw Paint

Sharing photo to public device

Image Gallery A+ Gallery

Unlock pattern with personal device

Pattern lock Smart app lock AppLock

Read document with different device

Text, scroll File Viewer Bible KJV

Edit text on different device

Editor Color note Text editor

slide-18
SLIDE 18

UI distribution time

  • It ranges from 132 to 735ms  Fast enough for interactive use

18

slide-19
SLIDE 19

Network transfer over time

  • Comparing transfer pattern of FLUID and other mirroring techniques

– Under the same scenario that a user types destination (11 characters) into the search box UI of Naver map

19

Search box UI 11 characters typed

Vysor Chromecast

slide-20
SLIDE 20

Conclusion

  • Designed & implemented FLUID

– Separation between app logic & UIs – Evaluated with 20 legacy apps for 10 multi-surface scenarios

  • Expect FLUID to accelerate

development of creative and useful apps to provide novel UX

20

slide-21
SLIDE 21

Thank you!

Visit cps.kaist.ac.kr/fluid for more information:)

21