Multi-touch Technology 6.S063 Engineering Interaction Technologies - - PowerPoint PPT Presentation

▶
multi touch technology
SMART_READER_LITE
LIVE PREVIEW

Multi-touch Technology 6.S063 Engineering Interaction Technologies - - PowerPoint PPT Presentation

Feb 09, 2023 252 likes •1.23k views

Multi-touch Technology 6.S063 Engineering Interaction Technologies Prof. Stefanie Mueller | MIT CSAIL | HCI Engineering Group how does my phone recognize touch? and why the do I need to press hard on airplane screens how would you build a

slide-1
SLIDE 1

Multi-touch Technology

6.S063 Engineering Interaction Technologies

  • Prof. Stefanie Mueller | MIT CSAIL | HCI Engineering Group
slide-2
SLIDE 2

how does my phone recognize touch? and why the… do I need to press hard on airplane screens…

slide-3
SLIDE 3

how would you build a multi-touch device?

  • which hardware do you use?
  • how does it work?

<2 minute brainstorming>

draw some sketches!

slide-4
SLIDE 4

resistive capacitive camera-based […]

types of touch technology::

there are lots of different…

slide-5
SLIDE 5

before we look at all of these, let’s zoom out a bit…

slide-6
SLIDE 6

before touch…

slide-7
SLIDE 7

in which year was the first touch screen invented?

<30s brainstorming>

slide-8
SLIDE 8

1986: Sensor Frame (McAvinney)

slide-9
SLIDE 9

Steve Jobs, 2007: “And we have invented a new technology called multi-touch, which is phenomenal. [0:33:33]

slide-10
SLIDE 10

but there is tech close to multi-touch that actually was invented even earlier…

slide-11
SLIDE 11

1963: Ivan Sutherland’s Light Pen (as part of SketchPad)

slide-12
SLIDE 12

1963: Ivan Sutherland’s Light Pen (as part of SketchPad)

slide-13
SLIDE 13

we have come a long way since then…

slide-14
SLIDE 14

30 years later, multi-touch has reached the consumer market…

slide-15
SLIDE 15

and then there’s still stuff that hasn’t reached the consumer market yet

slide-16
SLIDE 16

1991: Pierre Wellner, Digital Desk

slide-17
SLIDE 17

1991: Pierre Wellner, Digital Desk

slide-18
SLIDE 18

multi-touch:

engineering principles

slide-19
SLIDE 19

laser light plane (LLP) camera based:

slide-20
SLIDE 20

how does this recognize touch?

<30s brainstorming>

slide-21
SLIDE 21

laser light plane (LLP)

  • laser light shines as close as possible above the surface
  • when finger hits light plane, finger lights up
  • you can see this as bright spots in the camera image
slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

easy to do computer vision tracking based on this

slide-25
SLIDE 25

frustrated total internal reflection camera based:

slide-26
SLIDE 26

frustrated total internal reflection (FTIR)

slide-27
SLIDE 27

exiting light = bright blobs

frustrated total internal reflection (FTIR)

slide-28
SLIDE 28

frustrated total internal reflection (FTIR)

  • light is inserted into the sides of acrylic panel
  • light internally reflects because of FTIR phenomena
  • when finger touches panel, light gets ‘frustrated’
  • it escapes internal reflection and scatters downwards
  • you can see this as bright spots in the camera image
slide-29
SLIDE 29

exiting light = bright blobs

compliant surface

  • ptional: compliant surface
  • silicone rubber layer
  • improves dragging
  • acrylic doesn’t allow fingers to slide well, silicone does
  • improves sensitivity of the device
  • otherwise you need to press very hard
slide-30
SLIDE 30

exiting light = bright blobs

with compliant surface without compliant surface

slide-31
SLIDE 31

exiting

projection surface

  • ptional: projection surface
  • allows to display an image on the touch surface
  • can be made of e.g.paper, mylar, vellum, rosco grey
slide-32
SLIDE 32

exiting

projection surface

<30 second brainstorming>

if you want to project images onto your device, which type of LEDs do you need to use?

slide-33
SLIDE 33

exiting

projection surface infrared LEDs because otherwise your injected light for finger tracking

  • verlays with your projected content
slide-34
SLIDE 34

exiting

projection surface

<30 second brainstorming>

and what does that mean for the camera?

slide-35
SLIDE 35

exiting

infrared LEDs visible light projector infrared camera

slide-36
SLIDE 36

[Jeff Han, 2006]

slide-37
SLIDE 37

[Jeff Han, 2006] UIST 2005 paper (just got lasting impact award)

slide-38
SLIDE 38

Steve Jobs, 2007: “And we have invented a new technology called multi-touch, which is phenomenal. [0:33:33]

slide-39
SLIDE 39

this is pset1!

slide-40
SLIDE 40

rear diffused illumination (rear DI) camera based:

slide-41
SLIDE 41

<30 second brainstorming>

how does it work? how does the camera image look like? white or black spots?

slide-42
SLIDE 42

rear diffused illumination (rear DI)::

  • light shined from below the touch surface
  • a diffuser is placed on top of the touch surface
  • when the light hits a finger, light is reflected downwards
  • appears as bright blob in the camera image
  • same as FTIR, just light comes from below
slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45

<30 second brainstorming>

what can rear diffuse illumination detect that FTIR cannot? FTIR rear-DI mh, so the result the same then?

slide-46
SLIDE 46

FTIR vs.

  • nly detects objects

in direct contact with surface (light bounces inside sheet) can detect objects hovering over the surface (light reaches above sheet)

rear-DI

slide-47
SLIDE 47

front diffused illumination (front DI) camera based:

slide-48
SLIDE 48

rear DI front DI light from below light from above

slide-49
SLIDE 49

front DI

<30 second brainstorming>

how do we expect the camera image to look like?

slide-50
SLIDE 50

front DI finger blocks the light from the camera = fingers are black

slide-51
SLIDE 51

front diffused illumination (front DI)::

  • light shined from above the touch surface
  • a diffuser is placed on top of the touch surface
  • when a finger touches, a shadow is created underneath
  • appears as black blob in the camera image
slide-52
SLIDE 52
slide-53
SLIDE 53

[MTBiggie]

slide-54
SLIDE 54

infrared touch panels (ITP)

  • ptical (sensor based):
slide-55
SLIDE 55
slide-56
SLIDE 56

infrared touch panels (ITP)

  • infrared LEDs and light sensors
  • placed in a grid on bezel
  • LEDs transmit light to light sensors on the other side
  • anything that disrupts light, will register as touch
slide-57
SLIDE 57

1986: Sensor Frame (McAvinney)

slide-58
SLIDE 58

2011: ZeroTouch

slide-59
SLIDE 59

resistive touch panels (RTP) electric:

slide-60
SLIDE 60
slide-61
SLIDE 61

resistive touch panels (RTP)

  • the top and bottom sheet are conductive
  • they have a gap in-between, no electricity flowing
  • when the top sheet gets pressed by a finger, the

pressed point makes contact with the bottom sheet

  • electricity now get conducted at the contact point
slide-62
SLIDE 62
  • this is why in airplanes you have to push so hard…
slide-63
SLIDE 63

<30 second brainstorming>

how do we know where the user touches the screen?

slide-64
SLIDE 64

same principles as for the infrared touch panel x-y grid top layer: all horizontal lines bottom layer: all vertical lines when contact is made

  • nly these two line

conducts electricity resistive:

slide-65
SLIDE 65
  • lowest cost
  • low power consumption
  • work with finger, stylus, glove

benefits::

  • poor response to light touch…
  • dragging…
  • 26% of the market
slide-66
SLIDE 66

projected capacitance (PCAP)

slide-67
SLIDE 67
slide-68
SLIDE 68

resistive:

again same principle

capacitive:

slide-69
SLIDE 69

projected capacitance (PCAP)

  • 2 parallel conductive layers with grid lines
  • continues scanning of x/y grid lines (‘always on’)
  • grid lines create electro static field
  • when finger touches, the change in the electrodes

can be detected

slide-70
SLIDE 70
slide-71
SLIDE 71

2007: ‘we invented a new technology’ this is what your iphone uses…

slide-72
SLIDE 72

2001 SmartSkin: capacitive, no camera

slide-73
SLIDE 73

CHI 2002

slide-74
SLIDE 74

projected capacitance (PCAP)

  • no pressure force needed for detection
  • susceptible to electrical noise
  • more expensive than resistive
  • smart phones, tablets etc.
  • 64% of the market
slide-75
SLIDE 75

surface acoustic waves (SAW)

slide-76
SLIDE 76

resistive: capacitive: surface acoustic:

slide-77
SLIDE 77

surface acoustic waves (SAW)

  • basically the same as everything else just with sound
  • fingers in path absorb sound
  • thus you can detect them with a microphone
slide-78
SLIDE 78

<30 second brainstorming>

there are situations in which this grid based approach cannot correctly detect a finger’s position. how do you have to place two fingers to make it fail?

slide-79
SLIDE 79

it leads to ghosting! (camera-based setups don’t have this problem)

slide-80
SLIDE 80

moving forward…

slide-81
SLIDE 81

detecting pressure from touch…

slide-82
SLIDE 82

2005 GelForce

slide-83
SLIDE 83

<30 second brainstorming>

how does it work?

slide-84
SLIDE 84
slide-85
SLIDE 85
slide-86
SLIDE 86

CHI 2005

slide-87
SLIDE 87

UnMousePad

slide-88
SLIDE 88

SIGGRAPH 2009

slide-89
SLIDE 89

user identification on each touch

slide-90
SLIDE 90

what if we had finger print detection on the entire screen?

<30 second brainstorming>

slide-91
SLIDE 91
slide-92
SLIDE 92

UIST 2013

slide-93
SLIDE 93

let’s zoom out

slide-94
SLIDE 94

1986 2007 1963

towards more natural user interaction! use your hands to interact.

slide-95
SLIDE 95

let’s take a 5 minute break!

slide-96
SLIDE 96

end.