Okuli : Extending Mobile Interaction Through Near-Field Visible - - PowerPoint PPT Presentation

Okuli: Extending Mobile Interaction Through Near-Field Visible Light Sensing

Chi Zhang, Joshua Tabor, Jialiang Zhang and Xinyu Zhang Department of Electrical and Computer Engineering University of Wisconsin-Madison

Touch is a dominant mode of mobile interaction

But on-screen touch input is not always effective!

Screen multiplexed between display and input

Wastes precious display area On-screen keyboard hard to use

Input area depends on device size

Infeasible on wearable devices

Lack of physical interaction

No accurate feedback Separate device means extra burden

Can be solved by separating display and input

With passive wireless sensing

Bridging VLC and touch sensing

Previous solutions Array of LED/PD pairs: energy hungry, cumbersome Computer vision: heavy computation, obtrusive camera Machine-learning: excessive run-time training

Use PD/LED pairs in a different way

No phase information Amplitude is fine-grained and deterministic

Visible light channel

Requires a fine-grained model to achieve localization

Use PD/LED pairs in a different way

Unlike simple “finger blocking beam” model, fine-grained propagation model can enable lightweight localization With such model and 2 channels, we can locate user's finger

– This is how Okuli works

Okuli: overview

finger left PD right PD mobile device (e.g. smartphone) Workspace LED

Okuli: light grooming

2D localization → want to limit to 2D surface → light grooming

– Eliminates interferences from outside the surface

Hand Finger

surface PD FoV

Okuli: light grooming

Can be done with tiny lenses attaches to PDs / LED

Okuli: light grooming

For prototyping we use a 3D-printed shroud

LED left sensor right sensor

Okuli: light grooming

Horizontal Vertical

0.2 0.4 0.6 1 0.8 0.2 0.4 0.6 1 0.8

Before After

Okuli: channel model

Received signal is affected by multiple factors

– Factory calibration measures invariant part

← Angular response Angular response →

finger PD LED

Okuli: channel model

Received signal is affected multiple factors

– Model calculates variant part

Propagation loss → ← Propagation loss ← Finger reflectivity

finger PD LED

Okuli: channel model

Path loss is not simple: it is not actually only 2D

– Further away, more area visible – Model needs to compensate

surface PD FoV

Okuli: channel model

Finger reflectivity can be hard to characterize

– Abstract by interacting ratio of the beam – Overall reflectivity corrected by calibration

interacting non- interacting

incident reflect

Finger

Okuli: interference canceling

Surrounding light sources

– Can be much stronger than desired RSS – Not “coherent” with our light emission

Modulate our own emission with OOK

– Also helps saving energy

Okuli: interference canceling

Spatial solution: narrow vertical FoV Temporal solution: dynamic estimation & removal

– Identifies and tracks background – Also detects clicks

Background reflection

– Cannot be removed by modulation – Usually slow-changing and not very strong

Okuli: interference canceling

0.2 0.4 0.6 0.8 1 1 2 RSS Location Dark room Fluorescence light Diffusive sunlight Direct sunlight Without Cancellation With Cancellation 0.05 0.1 0.15 0.2 1 2 RSS Location No Background White Paper Static Background Dynamic Background Without Cancellation With Cancellation

Ambient light Dynamic background

Effective in most cases

Okuli: localization

For each point, model produces an expected RSS Samples are compared with these RSS Location that has minimum RSS error is selected

Prototyping Okuli

3D-printed shroud controls FoV Arduino drives LED and samples PDs Bluetooth connects Okuli to mobile devices Mobile device runs the algorithm

Performance

0.2 0.4 0.6 0.8 1 0.5 1 1.5 2 2.5 3 CDF Error (cm) Black paper White paper Glass 0.2 0.4 0.6 0.8 1 0.5 1 1.5 2 2.5 3 CDF Error (cm) Before After

Accuracy across different surfaces Accuracy across time (10 days)

Okuli is consistent across different surfaces and over time

Performance

20 40 60 80 100 1 2 3 4 5 6 7 Accuracy (%) User

90.7% 89.4% 94.1% 87.5% 93.8% 90.4% 91.8%

Keypad (20 keys) Okuli is consistent across different users

Performance

2 4 6 8 2 4 6 8 Y (cm) X (cm) Touchscreen Okuli 20 40 60 80 100 1 2 3 Accuracy (%) User Okuli

90.00% 91.50% 90.50%

Touchscreen

95.00% 93.60% 95.20%

Handwriting recognition Sample trackpad trace Okuli's performance is comparable with capacitive touch screens

Performance

Most energy cost by light emission

– Can duty-cycle to reduce

Processing costs very little

– Smooth UI, good user experience

100 200 300 400 0.1 0.2 0.3 0.4 0.5 Power Consumption (mW) LED CPU ADC Duty Cycle

Conclusion

• Fine-grained light propagation model can enable accurate near-

field visible light localization

• Multiple types of interferences exists in the visible light channel,

and can be effectively canceled

• Visible light channel allows us to achieve centimeter grade

passive localization with a compact system

Thank you!