Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. - - PowerPoint PPT Presentation

▶
mensch maschine interaktion 2 interactive environments
SMART_READER_LITE
LIVE PREVIEW

Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. - - PowerPoint PPT Presentation

Sep 04, 2023 251 likes •797 views

Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. Andreas Butz, Dr. Julie Wagner 1 LMU Mnchen Medieninformatik Andreas Butz Mensch-Maschine-Interaktion II WS2013/14

slide-1
SLIDE 1

Mensch-Maschine-Interaktion 2 Interactive Environments

  • Prof. Dr. Andreas Butz, Dr. Julie Wagner

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 1

slide-2
SLIDE 2

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 2

context and task challenges input technologies challenges in interaction design

  • utput technologies

Interactive Environments

slide-3
SLIDE 3

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Display: Front Projection

  • what we are doing here in class
  • simplest way to produce visual
  • utput on any surface
  • pro:

– cheap, simple – even light distribution – no additional space needed – space for legs under the table

  • contra

– interacting hand and person cast a shadow – only feasible for tabletops when firmly mounted

3

image source: http://www.rosco.com/

slide-4
SLIDE 4

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Display: Rear Projection

  • Pro:

– projector is hidden, space in front empty – no shadowing of the surface

  • Contra:

– Can only be done with space behind – complex mirror construction for tabletops – can create „hot spot“ with cheap screen

4

image source: http://www.rosco.com/

slide-5
SLIDE 5

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Display: Projection from the side ;-)

  • PlayAnywhere, Andy Wilson (Microsoft Research), 2005
  • Uses commercial short throw projector for front

projection at an angle of 40 degrees

  • Uses cameras for sensing

– mounted off axis from the projection – can see shadows caused by front projection – can recognize fingers and markers

  • Turns any flat surface (e.g., table)

into an interactive surface

5

slide-6
SLIDE 6

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 6

slide-7
SLIDE 7

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Display: Screens

  • What we initially used in our tabletop research @LMU

– High resolution and contrast + great color – Insensitive to ambient light – Can be bought with touch overlay for sensing

7

slide-8
SLIDE 8

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 8

FogScreen

  • A wall sized,

immaterial projection display

  • Image projected on

dry, non-hazardous fog (pure water)

  • Inventor: Ismo

Rakkolainen

slide-9
SLIDE 9

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 9

FogScreen (2)

  • Fog sandwiched

between airstreams

  • Immaterial

user can reach or walk through unhindered

slide-10
SLIDE 10

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 10

FogScreen (3)

  • Technical Details

– First introduced at UIST 2004 – So far only 9 FogScreens in the world – Price: 100k $ – Weight: around 100 kg – Needs: 10L/h, 300W

slide-11
SLIDE 11

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 11

FogScreen (4)

  • Pseudo 3D

– Double sided projection possible – E.g. opposite views of the same scene

slide-12
SLIDE 12

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 12

FogScreen(4)

  • Pseudo 3D – HeadTracking (S. DiVerdi)

– Tracking the users Head – Input used for adapting the frustum for accurate perspective rendering

slide-13
SLIDE 13

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 13

Everywhere Display Projector (IBM)

http://www.research.ibm.com/ed/

Claudio Pinhanez

slide-14
SLIDE 14

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 14

Everywhere display (cont.)

Output: a projector and a rotating mirror Input: a camera for interaction, NOT for image rectification!

slide-15
SLIDE 15

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 15

Undistorting the projected image

Place original image in the 3D model Virtual camera image shows it distorted Project the distorterd image from 3D model with the Real projector into the real world

– Distortions cancel each other out IF virtual camera and real projector are in the same location

slide-16
SLIDE 16

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 16

context and task challenges input technologies challenges in interaction design

  • utput technologies

Interactive Environments

slide-17
SLIDE 17

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Introduction and History

  • Early Research

– The MIT MetaDesk – Pierre Wellner‘s Digital Desk

17

slide-18
SLIDE 18

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Historic Interactive Surfaces

  • read http://www.billbuxton.com/

multitouchOverview.html !

  • early experiments with multi touch in the 1980ies
  • For this lecture: 2 prominent historic examples:

– Pierre Wellner‘s Digital Desk – MIT MetaDesk

18

slide-19
SLIDE 19

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Pierre Wellner‘s Digital Desk

  • Working prototype in 1991
  • Regular table with top projection
  • Overhead camera to detect fingers
  • Camera can also scan paper on the desk
  • Interaction with printed paper and digital

applications on the same surface

19

slide-20
SLIDE 20

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 20

slide-21
SLIDE 21

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

The MIT MetaDESK

  • Platform for exploring Tangible UIs (Ullmer & Ishii, 1997)
  • Also uses top projection
  • Various projects built on top of it

21

slide-22
SLIDE 22

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Problems and Particularities

  • Asymmetric bimanuality
  • Territoriality on tables
  • Direction and orientation on tables
  • Midas Touch Problem
  • Occlusion Problem
  • Fat Finger problem

22

slide-23
SLIDE 23

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Territoriality on tables (Scott 2004)

  • Studies on how people use the space on a table

– puzzle, game, Lego activities + room planning on round tables

  • Different areas on the table surface

– personal space (directly in front of person) – group space (reachable by all members) – storage space (in the periphery)

  • Boundaries between areas are flexible

23

slide-24
SLIDE 24

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Territoriality on tables (Scott 2004)

24

slide-25
SLIDE 25

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Territoriality on tables (Scott 2004)

  • Design Implications:

– Provide visibility and transparency of action – Provide appropriate table space – Provide functionality in the appropriate locality – Allow casual grouping of items and tools in the workspace

25

slide-26
SLIDE 26

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Orientation on tables (Kruger 2003)

  • Basic problem: no clearly defined „up“ direction when

interacting with multiple users around a table

  • Known approaches:

– Fixed orientation – Manual orientation – Person-based automatic orientation – Environment-based automatic orientation

26

ABC ABC

slide-27
SLIDE 27

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Orientation on tables (Kruger 2003)

  • Orientation can serve as a collaborative resource:

– Using someone else’s alignment conveyed support – Orientation could establish the intended audience – Orientation was also used to create a personal space.

27

slide-28
SLIDE 28

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Orientation on tables (Kruger 2003)

  • 3 main roles of orientation:
  • Comprehension

– Ease of reading – Ease of task – Alternate perspective

  • Coordination

– Establishment of personal spaces – Establishment of group spaces – Ownership of objects

  • Communication

– Intentional communication – Independence of orientation

28

slide-29
SLIDE 29

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Midas Touch Problem

  • General problem with touch input:

– positioning and selection is the same – cannot position the pointer without selecting – there is no „hover“ state

  • Name goes back to king Midas

– everything he touched turned to gold – ...also food, so he was in danger to starve...

  • Straightforward solution: dwell time

– trigger a function only after a certain time – introduces artificial delays, disturbs the flow

  • Also well-known in gaze-based interaction (eye tracking)

29

http://de.wikipedia.org/wiki/Midas

slide-30
SLIDE 30

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Dealing with the Midas Touch Problem

  • Lift-off techniques (e.g., Potter et al. CHI 1988)

– cursor is offset from the finger and displayed during exploration – function is only triggered when the finger leaves the surface – can be observed in the video on „shift“, earlier in this class

  • Pressure sensing (e.g., Richter et al. AutoUI 2010)

– finger can explore the screen in hover state – selection by pressing harder

30

slide-31
SLIDE 31

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Interaction Techniques on Int. Surfaces

  • Motivation: thinking about bumptop
  • Single touch

– RNT

  • Dual touch

– The Pinch etc.

  • Multi touch

– possible contradictions

  • Shape-based

– Bringing physics to the surface

  • Tangible UIs on surfaces

– URP, illuminating light, SLAP, eLabBench

31

slide-32
SLIDE 32

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Bumptop - the original video

32

slide-33
SLIDE 33

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Making fun of Bumptop - discussion

33

slide-34
SLIDE 34

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

RNT - Rotate and Translate with 1 Touch Point

  • Problem: only 1 touch point (= mouse)
  • Goal: rotate and translate in a single gesture
  • Idea: use a physical model (inertia, friction)

– friction force is opposite to the direction of movement – rotation is recalculated in every frame

34

slide-35
SLIDE 35

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Rotate, Translate and Scale with 2 Touch Points

  • track 2 points from frame to frame
  • compute scaling from change in distance d to d‘
  • compute rotation from angle between <d,d‘>
  • compute translation t and use directly

35

d d‘ d t

slide-36
SLIDE 36

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Possible Contradictions with >2 Touch Points

  • Consider R+T+S method for 2 TP
  • With 3 TP we can find 3 pairs
  • they will almost certainly yield

– different d, d‘ – different orientation – possibly even different translations

  • How to deal with this?

– ignore 1 point ;-) looks weird in certain cases – use mean R,T,S to minimize error – deform the underlying object

36

d d‘ d t

slide-37
SLIDE 37

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Shape-based interaction

37

  • Interaction in the real world uses

not just contact points

  • We use whole hands, arms, tools
  • Cannot be adequately expressed

using just contact points

  • How can we deal with this?
  • Remember the lava lamp in Jeff

Han‘s TED talk?

  • Seriously: How can we do useful

stuff with this?

slide-38
SLIDE 38

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Tangible UIs on Interactive Surfaces

  • classic TUI project: URP
  • another classic: illuminating light
  • SLAP widgets
  • Other examples already seen (hence not repeated):

– MetaDesk – PhotoHelix

38

slide-39
SLIDE 39

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 39

Luminous room: Urban Planning (URP)

(John Underkoffler and Hiroshi Ishii, CHI 99)

  • Move physical models of

houses on a desk surface

  • Simulate in the computer:

– Shadows – Window reflections – Air flow and wind

slide-40
SLIDE 40

40

slide-41
SLIDE 41

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 41

Luminous room: Illuminating Light

(John Underkoffler and Hiroshi Ishii, CHI 98)

  • Simulation of optical/holographic setups
  • Phys. objects represent optical elements
  • Top projection of resulting laser beam
slide-42
SLIDE 42

42

slide-43
SLIDE 43

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 43

SLAP Widgets [Weiss et al. CHI 2009]

  • Transparent Tangible objects for interactive surfaces
  • Use optical sensing (FTIR + DI)
  • Can be augmented with digital content
  • Idea: flexible multi-purpose tangible widget set
  • SLAP Widget set:

– a) Keypads. – b) Knob. – c) Slider. – d) Keyboard.

slide-44
SLIDE 44

44

slide-45
SLIDE 45

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 45

eLabBench [Tabard et al., ITS 2011 best paper]

  • Idea: make a Biology lab bench interactive
  • Use the same lab instruments as before
  • Augment them with digital information
slide-46
SLIDE 46

46

slide-47
SLIDE 47

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Multi Display Interaction

  • Historic: Augmented Surfaces, Pick&drop
  • newer: bump, stitch
  • recent example: Touch projector

47

slide-48
SLIDE 48

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Augmented Surfaces

Jun Rekimoto and Masanori Saitoh, CHI'99

  • Combination of mobile devices

and projection surfaces

  • Interaction techniques:

–hyperdragging –pick-and-drop –pick-and-beam –digital attachment

  • interaction objects for tangible interaction
  • Camera-based acquisition of images
  • Selection from physical catalogues

Video

48

slide-49
SLIDE 49

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide 49

slide-50
SLIDE 50

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

TouchProjector [Boring et al., CHI `11]

50

slide-51
SLIDE 51

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Stitching & Bumping [Hinckley et al.]

51

slide-52
SLIDE 52

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Environment context and task challenges input technologies challenges in interaction design

  • utput

technologies

Take-away Message

  • many visual output technologies
  • displays
  • tabletops
  • (steerable) projection
  • instrumented environment = many displays
  • think about coordination
  • interaction techniques need to be invented
  • No clear interaction concept yet
  • every device works differently
  • think about rules that can be applied to all
  • field of ongoing research!!

52

slide-53
SLIDE 53

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Additional Literature

  • Rotation and translation mechanisms for tabletop interaction.

Mark S. Hancock, Frédéric Vernier, Daniel Wigdor and Sheelagh Carpendale, and Chia Shen. In Proc. Tabletop, pp. 79-86, 2006.

– (also the source of the image on slide 25)

  • A. D. Wilson, S. Izadi, O. Hilliges, A. Garcia-Mendoza, D. Kirk:

„Bringing Physics to the Surface“, ACM UIST 2008 - Best Paper Award

  • Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings,

Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In CHI '09:

  • A. Tabard, J.D. Hincapié-Ramos, M. Esbensen, and J.E.

Bardram: „The eLabBench: An Interactive Tabletop System for the Biology Laboratory“. ACM ITS 2011 - Best Paper Award

53

slide-54
SLIDE 54

LMU München — Medieninformatik — Andreas Butz — Mensch-Maschine-Interaktion II — WS2013/14 Slide

Environment context and task challenges input technologies challenges in interaction design

  • utput

technologies

Michel Beaudouin-Lafon: Instrumental Interaction in Multisurface Environments

54

February 5th, 2014 10:15 - 11:45 here in this room