[PPT] - Body-Mounted Cameras Claudio Fllmi foellmic@student.ethz.ch May PowerPoint Presentation

SLIDE 1

May 28, 2013 Distributed Systems Seminar 1

Body-Mounted Cameras

Claudio Föllmi

foellmic@student.ethz.ch

SLIDE 2

May 28, 2013 Distributed Systems Seminar 2

Outline

Google Glass
EyeTap
Motion capture
SenseCam

SLIDE 3

May 28, 2013 Distributed Systems Seminar 3

Introduction

Cameras have become small, light and cheap
We can now wear them constantly
So what new things can we do with them?

[ h t t p : / / w w w . t h a i e a s y e l e c . n e t / i n d e x . p h p / S e n s

r

s / L i g h t

/
I

m a g i n g / 4 d s y s t e m

J

P E G

C
l
r
C

a m e r a

U

A R T

T

T L

I

n t e r f a c e

O

V 5 2 8

C

h i p / p _ 1 7 7 . h t m l ]

SLIDE 4

May 28, 2013 Distributed Systems Seminar 4

Google Glass

[ h t t p : / / c

m

m

n

s . w i k i m e d i a .

r

g / w i k i / F i l e : G

g

l e _ G l a s s _ E x p l

r

e r _ E d i t i

n

. j p e g ]

SLIDE 5

May 28, 2013 Distributed Systems Seminar 5

Google Glass

Like a head-worn smartphone
Sensors, wi-fi, bluetooth, camera
Input over voice commands, touchpad or phone app
Runs Android
App support

SLIDE 6

May 28, 2013 Distributed Systems Seminar 6

Schematic

[ U S p a t e n t a p p l i c a t i

n

2 1 3 4 4 4 2 ]

SLIDE 7

May 28, 2013 Distributed Systems Seminar 7

Applications

Lifelogging
Looking up information on the internet
Remote-controlling smartphone
Getting notifications from smartphone

SLIDE 8

May 28, 2013 Distributed Systems Seminar 8

Too futuristic?

[ p l u s . g

g

l e . c

m

/ 1 5 8 4 7 6 7 4 8 1 8 3 7 2 7 7 / p h

t
s

]

SLIDE 9

May 28, 2013 Distributed Systems Seminar 9

Meet Thad Starner and Steve Mann

[ h t t p : / / w w w . h i s t a n c i a . c

m

/ w e a r a b l e

c
m

p u t i n g / ]

SLIDE 10

May 28, 2013 Distributed Systems Seminar 10

Steve Mann over the years

[ M a n n 2 4 ]

SLIDE 11

May 28, 2013 Distributed Systems Seminar 11

EyeTap

[ M a n n 2 1 2 ]

SLIDE 12

May 28, 2013 Distributed Systems Seminar 12

Schematic

[ M a n n 2 1 3 ]

SLIDE 13

May 28, 2013 Distributed Systems Seminar 13

How it works

Incoming light is diverted onto camera
Captured image is processed
Processed image is projected into eye
Functionally, the eye becomes a camera
To the outside, the camera replaces the eye

SLIDE 14

May 28, 2013 Distributed Systems Seminar 14

Applications

Lifelogging
Capture events from exact perspective of user
Augmented / mediated reality
Latency between capture and projection will disturb

users (simulation sickness)

Correction of visual impairment
Not just long-/shortsightedness
Increase visual dynamic range

SLIDE 15

May 28, 2013 Distributed Systems Seminar 15

HDR filter

[ " R e a l t i m e H D R ( H i g h D y n a m i c R a n g e ) V i d e

f
r

E y e T a p W e a r a b l e C

m

p u t e r s , F P G A

B

a s e d S e e i n g A i d s , a n d G l a s s E y e s " , M a n n e t a l , 2 1 2 ]

SLIDE 16

May 28, 2013 Distributed Systems Seminar 16

Color blindness

About 8% of all men are

red-green colorblind

Much of our everyday life is

color-coded

EyeTap could automatically

replace colors, or label colored surfaces

[ h t t p : / / e n . w i k i p e d i a .

r

g / w i k i / C

l
r

b l i n d n e s s ]

SLIDE 17

May 28, 2013 Distributed Systems Seminar 17

Comparison with Glass

EyeTap

More potential for

mediated reality

Can replace prescription

glasses Glass

Standard smartphone

functionality

Compact enough be worn

as an accessory

SLIDE 18

May 28, 2013 Distributed Systems Seminar 18

Future development

Size, weight and battery life will improve through normal

technical progress

More computing power will allow for more complex

applications

Latency is limiting factor of mediated reality
Society's reaction to Glass will be very important
Broad acceptance of Glass will make EyeTap more

acceptable

SLIDE 19

May 28, 2013 Distributed Systems Seminar 19

What if we film the user instead?

SLIDE 20

May 28, 2013 Distributed Systems Seminar 20

Motion capture

[ h t t p : / / a v a t a r b l

g

. t y p e p a d . c

m

/ a v a t a r

b

l

g

/ 2 1 / 5 / b e h i n d

t

h e

s

c e n e s

l
k
a

t

t

h e

m
t

i

n
c

a p t u r e

t

e c h n

l
g

y

u

s e d

i

n

a

v a t a r . h t m l ]

SLIDE 21

May 28, 2013 Distributed Systems Seminar 21

Motion capture

We have already seen a

system using body-mounted cameras for capturing full body motion

(April 9, talk by Antoine

Kaufmann)

[ S h i r a t

r

i 2 1 1 ]

SLIDE 22

May 28, 2013 Distributed Systems Seminar 22

Approach

Capture the face using one camera and multiple light

sources (Photometric stereo)

Can be combined with other systems
Many systems have a normal camera pointed at the

face anyway

The actor should be able to move freely
Ambient light changes

SLIDE 23

May 28, 2013 Distributed Systems Seminar 23

Why capture the face separately?

Face is the most expressive part of body language
Even small mistakes will stand out
Traditionally, artists would correct generated images by

hand

SLIDE 24

May 28, 2013 Distributed Systems Seminar 24

How it works

[ J

n

e s 2 1 1 ]

SLIDE 25

May 28, 2013 Distributed Systems Seminar 25

How it works

Three different lighting directions
One with ambient lighting only
Capture 120 fps input for 30 fps output
Change lights at 360Hz to eliminate flickering

[ J

n

e s 2 1 1 ]

SLIDE 26

May 28, 2013 Distributed Systems Seminar 26

How it works

Assumes lambertian

reflectance

Surface will look

equally bright from a range of angles

[ h t t p : / / c

m

m

n

s . w i k i m e d i a .

r

g / w i k i / F i l e : L a m b e r t 2 . g i f ]

SLIDE 27

May 28, 2013 Distributed Systems Seminar 27

Equation

I = L * N A

I: image intensity (measured by camera)
L: lighting direction (3x3 matrix, known)
N: surface normal (vector that we want to know)
A: albedo (scaling factor)
Surface geometry can then be reconstructed from

normals

SLIDE 28

May 28, 2013 Distributed Systems Seminar 28

Corrections

Closeness of both lights and camera leads to

inaccuracies

Lighting direction for each pixel depends on depth
Use a generic smoothed face to initialize lighting

directions

[ J

n

e s 2 1 1 ]

SLIDE 29

May 28, 2013 Distributed Systems Seminar 29

Video demonstration

[ J

n

e s 2 1 1

h

t t p : / / w w w . y

u

t u b e . c

m

/ w a t c h ? v = R G t F P L c i F L Q ]

“Head-Mounted Photometric Stereo for Facial Performance Capture” http://www.youtube.com/watch?v=RGtFPLciFLQ

SLIDE 30

May 28, 2013 Distributed Systems Seminar 30

Results

Shadow artifacts show up as white albedo around nose

[ J

n

e s 2 1 1 ]

SLIDE 31

May 28, 2013 Distributed Systems Seminar 31

Results

Subtle movements are captured
Exactly where non-camera-based approaches fail
Using infrared light leads to more artifacts
Computation of normals and geometry reconstruction can

be done in real-time

SLIDE 32

May 28, 2013 Distributed Systems Seminar 32

Comparison with more detailed method

Jones et al, 2011 Beeler et al, 2011

SLIDE 33

May 28, 2013 Distributed Systems Seminar 33

Comparison with more detailed methods

Actors can move freely
Can handle changes in

ambient light

Captures even small

wrinkles

Needs multiple cameras

and uniform lighting

Actors must hold head still

SLIDE 34

May 28, 2013 Distributed Systems Seminar 34

Next steps

Use a picoprojector for lighting
Use a customized input head model
Arrange lights further apart
Track shaking of camera
Try out more faces
Combine with other methods

SLIDE 35

May 28, 2013 Distributed Systems Seminar 35

Lifelogging

[ h t t p : / / w w w . v i c

n

r e v u e . c

m

/ i m a g e s / 9 3 . j p g ]

SLIDE 36

May 28, 2013 Distributed Systems Seminar 36

SenseCam

Automatically triggered camera
Lightweight
Unobtrusive
Battery life of at least 12 hours
Capture interesting moments

SLIDE 37

May 28, 2013 Distributed Systems Seminar 37

Internals

[ H

d

g e s 2 6 ]

SLIDE 38

May 28, 2013 Distributed Systems Seminar 38

How it works

Sensors judge if situation is

interesting

If yes, take picture
Take picture after timeout
Log reason why picture was

taken

Battery lasts over 24 hours

when taking a picture every 30 seconds

[ H

d

g e s 2 6 ]

SLIDE 39

May 28, 2013 Distributed Systems Seminar 39

Viewer application

[ h t t p : / / r e s e a r c h . m i c r

s
f

t . c

m

/ e n

u

s / u m / c a m b r i d g e / p r

j

e c t s / s e n s e c a m / r e v i e w . h t m ]

SLIDE 40

May 28, 2013 Distributed Systems Seminar 40

Applications

Lifelogging
Monitoring behavior (e.g. for scientific studies)
Measuring typical environment
Replace laboratory settings
Memory aid
Therapeutic tool for mental & cognitive disorders

SLIDE 41

May 28, 2013 Distributed Systems Seminar 41

Treatment for memory disorders

Many patients benefit from reading a diary or looking at

pictures of events they witnessed

But they are the least reliable group to write diaries and

take pictures

SenseCam does not need any attention during the event

it captures

SLIDE 42

May 28, 2013 Distributed Systems Seminar 42

Case study

63 year old married woman
Memory loss after illness
Limbic encephalitis (brain inflammation)
Forgets events completely after a week

SLIDE 43

May 28, 2013 Distributed Systems Seminar 43

Results

[ H

d

g e s 2 6 ]

SLIDE 44

May 28, 2013 Distributed Systems Seminar 44

Impact

Sparked interdisciplinary interest in memory aids
Original paper was cited over 300 times
Vicon manufactured a commercial version for

researchers

There is a conference just about SenseCam

SLIDE 45

May 28, 2013 Distributed Systems Seminar 45

Future

There will be more research on lifelogging and its effects
SenseCam devices will become broadly available to

normal consumers

Will become a standard gear for scientific studies
Many applications in therapy

SLIDE 46

May 28, 2013 Distributed Systems Seminar 46

Recap

We have seen three very different

uses of cameras

EyeTap replaces your eye with a

camera

SenseCam automatically takes

pictures for you

One camera is enough for motion

capture of faces

SLIDE 47

May 28, 2013 Distributed Systems Seminar 47

Outlook

We already all have a camera with us all the time
Soon, we will also have a camera ready all the time
And soon after, we will be recording all the time

[ h t t p : / / c

m

m

n

s . w i k i m e d i a .

r

g / w i k i / F i l e : H A L 9 . s v g ]

SLIDE 48

May 28, 2013 Distributed Systems Seminar 48

Questions?

SLIDE 49

May 28, 2013 Distributed Systems Seminar 49

Sources

“Continuous Lifelong Capture of Personal Experience with

EyeTap”, Steve Mann, CARPE'04, October 15, 2004

“Through the Glass, Lightly”, Steve Mann, IEEE Technology and

Society, Vol. 31, No. 3, Pages 10-14, 2012

“My 'Augmediated' Life”, Steve Mann, IEEE Spectrum, March 1,

2013

http://spectrum.ieee.org/geek-life/profiles/steve-mann-my-augmediated-life

“Head-mounted Photometric Stereo for Performance Capture”,

Andrew Jones, Graham Fyffe et al, 2011 Conference for Visual Media Production, Pages 158-164, 2011

“SenseCam: A Retrospective Memory Aid”, Steve Hodges et al,

Ubicomp 2006, Pages 177-193, 2006

SLIDE 50

May 28, 2013 Distributed Systems Seminar 50

Secondary sources

“High-Quality Passive Facial Performance Capture using

Anchor Frames”, Thabo Beeler et al, ACM Trans. Graph. 30, 4, Article 75 (July 2011)

“Physical assault by McDonald's for wearing Digital Eye