Interactive 3D Displays Shaban Shabani sshabani@student.ethz.ch - - PowerPoint PPT Presentation

▶
interactive 3d displays
SMART_READER_LITE
LIVE PREVIEW

Interactive 3D Displays Shaban Shabani sshabani@student.ethz.ch - - PowerPoint PPT Presentation

Apr 06, 2023 110 likes •577 views

Interactive 3D Displays Shaban Shabani sshabani@student.ethz.ch 14.05.2013 Distributed Systems Seminar 1 LightSpace BiDi HoloDesk Vermeer Interaction with displays Old technique, still in use Several drawbacks High

slide-1
SLIDE 1

Interactive 3D Displays

Shaban Shabani

sshabani@student.ethz.ch

14.05.2013 1 Distributed Systems Seminar

slide-2
SLIDE 2

Interaction with displays

§ Old technique, still in use

§ Several drawbacks § High adaption

§ Isn’t this already perfect? § Why do we need more?

14.05.2013 2 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-3
SLIDE 3

Touch screens

§ Displays show dynamic data

§ Heavy content § Requires efficient interaction by the user

§ Touch-screens

§ Direct interaction on the screen § Controlling content with great enthusiasm § Multi-touch feature § Wide application

§ Drawbacks

§ Single screen § Limited to the physical extent of the display

14.05.2013 3 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-4
SLIDE 4

Beyond touch screens

§ Move interactivity off the display and into environment § “Office of the future”

14.05.2013 4 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

“The office of the future: A unified approach to image-based modeling and spatially immersive displays”

  • R. Raskar et al.
slide-5
SLIDE 5

LightSpace – “smart room”

“Once, computers were the size of entire room. Today, the entire room is turned into a computer.”

§ Small room installation

§ Use of depth cameras and projectors § Interactions on, above and between surfaces

§ Merge of techniques

§ Surface computing § Augmented reality

14.05.2013 5 Distributed Systems Seminar

LightSpace configuration

LightSpace BiDi HoloDesk Vermeer

“Combining multiple depth cameras and projectors for interactions on, above and between surfaces” A.Wilson and H.Benko

slide-6
SLIDE 6

Overview

§ Depth-sensing cameras

§ Calculate depth of the objects in the scene § Track user’s position and interactions § Capture in real-time 3D mesh model

  • f the sensed space

§ Projectors

§ Virtual objects projected in the real space on top of real objects

14.05.2013 6 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-7
SLIDE 7

Interactive displays

§ Surface everywhere

§ Room physical surfaces should be interactive “displays” § e.g. interactive wall and interactive table

§ The room is the computer

§ Space between surfaces is active as well

§ Body as display

§ Graphics projected on user’s body

14.05.2013 7 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-8
SLIDE 8

Operation

§ Depth cameras provide 3D mesh model § Implementation of interactions

§ Avoids calculations directly

  • n 3D mesh model

§ Transform data to an image generated by a “virtual camera” § 3 orthographic virtual cameras “wall”, “table” and “plan”

14.05.2013 8 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-9
SLIDE 9

Interactions

§ Interactive surface

§ Manipulate with projected

  • bjects in the surfaces

§ Through-body transitions between surfaces

§ Move objects between surfaces

§ “Picking up” objects

§ Drag an object and pick it up with hand

14.05.2013 9 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-10
SLIDE 10

Video

14.05.2013 10 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

http://www.youtube.com/watch?v=gc_Xj7Z8aLU

slide-11
SLIDE 11

More than surfaces

§ The world is 3D § More DOF for tasks with such demands § More visual information § Exciting and more realistic than 2D

14.05.2013 11 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-12
SLIDE 12

3D displays

§ Require the wearing of special glasses § Non interactive § Interactive ones mainly require additional hardware

§ Data gloves § Head-mounted display § Gamming controllers

14.05.2013 12 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-13
SLIDE 13

BiDi Screen

§ Bidirectional screen § Turn an LCD in a thin display to support:

§ Image capture and display § On-screen 2D multi-touch § Off-screen 3D interaction

§ No need for glasses or HMD

14.05.2013 13 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

“BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields” M.Hirsch et al.

slide-14
SLIDE 14

BiDi - Inspiration

14.05.2013 14 Distributed Systems Seminar

Light sensitive display Depth camera Multi-touch display

§ Combine in a single device

LightSpace BiDi HoloDesk Vermeer

slide-15
SLIDE 15

BiDi - Challenges

14.05.2013 15 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

§ Build a thin portable device § Enable multi-touch and 3D gesture interaction § Collocated image capture and display § How to capture depth from an LCD?

slide-16
SLIDE 16

Background - Traditional touch?

14.05.2013 16 Distributed Systems Seminar

§ Resistive and capacitive multi-touch displays

§ Only sense the surface of the display ?

LightSpace BiDi HoloDesk Vermeer

slide-17
SLIDE 17

Motivation – Optical multi-touch device

14.05.2013 17 Distributed Systems Seminar

§ LCDs from Sharp and Planar § Optical sensors

§ Capture sharp image of objects when in contact with the surface of the screen § As objects move away from the screen, the images are blurred Display with embedded optical sensors

LightSpace BiDi HoloDesk Vermeer

slide-18
SLIDE 18

Motivation – Optical multi-touch device

14.05.2013 18 Distributed Systems Seminar

§ LCDs from Sharp and Planar § Optical sensors

§ Capture sharp image of objects when in contact with the surface of the screen § As objects move away from the screen, the images are blurred Display with embedded optical sensors

LightSpace BiDi HoloDesk Vermeer

slide-19
SLIDE 19

Design vision

14.05.2013 19 Distributed Systems Seminar

LCD , displaying mask Optical sensor array ~2.5 cm ~50 cm

LightSpace BiDi HoloDesk Vermeer

§ Separate the optical sensor by a small distance from the LCD § Collocated display and capture

slide-20
SLIDE 20

Pinhole camera

14.05.2013 20 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

§ Simple camera without lens and small aperture § Light passes through the single point § Projects inverted image on the opposite side of the box

slide-21
SLIDE 21

Design overview

14.05.2013 21 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

§ Display a pinhole of arrays

§ Creates tiny images on the sensor array each with different angle view

§ Camera captures images § Analyze patterns of the images

§ Decode depth Camera Pinhole array Diffuser

slide-22
SLIDE 22

Interactions

§ On-screen multi-touch interactions on objects

§ Resize § Rotate

§ Off-screen 3D gestures

§ Move objects in 4 directions § Zoom in/out objects in 2 directions

14.05.2013 22 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-23
SLIDE 23

Video

14.05.2013 23 Distributed Systems Seminar 14.05.2013 23 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

http://www.youtube.com/watch?v=kXuxK6IeQfo

slide-24
SLIDE 24

Summary

§ Benefits

§ A modified LCD that supports multi-touch and 3D gestures § No special hardware § Real-time interaction

§ Limitations

§ Size § Lighting

§ Requires external lighting § Not functional in case of absence of illumination

14.05.2013 24 Distributed Systems Seminar

LightSpace BiDi HoloDesk Vermeer

slide-25
SLIDE 25

In-volume 3D interaction

§ Direct interaction on the 3D content not possible

§ Behind glass § Some require special glasses and data gloves

14.05.2013 25 Distributed Systems Seminar

LightSpace HoloDesk BiDi Vermeer

slide-26
SLIDE 26

HoloDesk

§ An interactive Augmented Reality system § Interact with 3D virtual objects

§ ‘Inside’ the display § No need for body-worn hardware

§ Physically realistic interactions § ‘Walk-up-and-use’

§ No user instrumentation

14.05.2013 26 Distributed Systems Seminar

LightSpace HoloDesk BiDi Vermeer

“HoloDesk: Direct 3D Interactions with a Situated See-Through Display” O.Hilliges et al.

slide-27
SLIDE 27

Design overview

14.05.2013 27 Distributed Systems Seminar

§ Interaction volume

§ Seen through beam splitter

§ Optical see through mirror (Beam splitter)

§ Reflects light to the user from the LCD § Forms a virtual image on interaction volume

§ RGB Webcam

§ Tracks user’s head 6DOF

§ Kinect + mirror

§ Mirror fold the Kinect’s optic § Sense the interaction volume

LightSpace HoloDesk BiDi Vermeer

slide-28
SLIDE 28

Operation

14.05.2013 28 Distributed Systems Seminar

§ Correctly virtual graphics

§ Tracking and estimating 3D position of head § Continuously updating the rendering

§ Kinect

§ Real-time depth data § Occlusion § Inter-shadowing

Virtual scene Occlusion Inter-shadowing

LightSpace HoloDesk BiDi Vermeer

slide-29
SLIDE 29

Implementation

14.05.2013 29 Distributed Systems Seminar

§ Simulating human grasping

§ Accurately model collision and friction forces exerted onto virtual objects § Kinect depth data approximated by small spherical rigid bodies § Approximate the shape, motion and deformation of 3D physical objects § Model interpenetration of objects

LightSpace HoloDesk BiDi Vermeer

a) b) c) d)

slide-30
SLIDE 30

Applications

14.05.2013 30 Distributed Systems Seminar

§ Mix of real and virtual content § Gaming § Virtual prototype

§ Smartphone § Touch-enabled

§ Telepresence

§ Users share single virtual 3D scene § Interactions relayed to a remote unit

LightSpace HoloDesk BiDi Vermeer

slide-31
SLIDE 31

Interactions

14.05.2013 31 Distributed Systems Seminar

§ Physics-enabled interactions

§ Interact with virtual in realistic way

§ Rich free interactions

§ Juggling, grasping

LightSpace HoloDesk BiDi Vermeer

slide-32
SLIDE 32

Video

14.05.2013 32 Distributed Systems Seminar

LightSpace HoloDesk BiDi Vermeer

http://www.youtube.com/watch?v=JHL5tJ9ja_w

slide-33
SLIDE 33

Summary

14.05.2013 33 Distributed Systems Seminar

§ Benefits

§ Novel system which supports full 3D interactions § No user instrumentation § No head-worn sensors § Direct interactions with 3D objects § Rich physically inspired interactions § Various applications

§ Limitations

§ Does not provide full simulation of object motion in the interaction volume § Failures in case of occlusion of physical objects § Finger grasping requires visible fingertips

LightSpace HoloDesk BiDi Vermeer

slide-34
SLIDE 34

Towards 360-degree viewing displays

14.05.2013 34 Distributed Systems Seminar

§ Previous displays have user restrictions on interacting with volume

§ Limited viewpoint § Glass separates the physical display from user

LightSpace Vermeer BiDi HoloDesk

slide-35
SLIDE 35

Video

14.05.2013 35 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk

“Rendering for an Interactive 360º Light Field Display” A.Jones et al.

http://www.youtube.com/watch?v=YKCUGQ-uo8c

slide-36
SLIDE 36

Vermeer

14.05.2013 36 Distributed Systems Seminar

§ A new enabling technology for in-volume interaction § 360°viewable 3D display

§ Viewpoint corrected § No need for eyewear § No user instrumentation § Directly touch and interact with 3D objects inside the display volume

LightSpace Vermeer BiDi HoloDesk

“Vermeer: Direct interaction with a 360-degree viewable 3D display” A.Batler et al

slide-37
SLIDE 37

Motivation

14.05.2013 37 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk

§ Optical illusion using 2 parabolic mirrors

§ Object placed at the bottom of the mirror reimaged as it was real, above the unit § Provides 360°view § Allows free interaction without encumbering the projection

slide-38
SLIDE 38

Video - Parabolic mirrors

14.05.2013 38 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk

slide-39
SLIDE 39

Design overview

14.05.2013 39 Distributed Systems Seminar

§ An extended 360 viewable 3D display

§ Projection § Imaging from below

§ View restrictive spinning diffuser

§ Provides the narrow viewing angle § Different views to multiple users

§ High speed DMD projector

§ Displays multiple viewpoints of the 3D scene at high rates § 2880 images/second 192 different views per rotation

LightSpace Vermeer BiDi HoloDesk

slide-40
SLIDE 40

Sensing interactions using Kinect

14.05.2013 40 Distributed Systems Seminar

§ Kinect depth camera positioned above

§ User places finger inside the volume § Contour of the hand detected and fingertips tracked § Animate 3D scene

§ Physics simulation (nVidia PhysX)

§ More realistic interaction § Control visual objects with real-world concepts: forces, collisions and frictions

LightSpace Vermeer BiDi HoloDesk

slide-41
SLIDE 41

Sensing interactions using infrared

14.05.2013 41 Distributed Systems Seminar

§ Enabling interactions with Kinect

§ Increases complexity § Suffers from bulk and occlusion

§ Infrared reimaging

§ Parabolic mirror reimage light in the near IR-range § An object reimaged using IR, floats in the display area § Fingertips intersecting with objects detected with an IR camera

§ Sense intersections between physical

  • bjects above and IR illuminated inside

§ e.g. spinning sphere § Distinguish finger touching object from

  • thers

LightSpace Vermeer BiDi HoloDesk

slide-42
SLIDE 42

Video

14.05.2013 42 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk

http://www.youtube.com/watch?v=lW7k-6FUxoo

slide-43
SLIDE 43

Summary

14.05.2013 43 Distributed Systems Seminar

§ Benefits:

§ Novel technique in interacting with 360°viewable 3D displays § Supports multiple users § No need for glasses and instrumentation

§ Limitations

§ Small dimensions of the viewable volume § View constraints when viewpoint too high or too low § Distortion of the displayed image § Limited rendering fidelity

LightSpace Vermeer BiDi HoloDesk

slide-44
SLIDE 44

Summary

14.05.2013 44 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk Interactions on 2D surfaces 3D off-screen interactions In-volume interactions 360° viewable display

slide-45
SLIDE 45

Summary

14.05.2013 45 Distributed Systems Seminar

LightSpace Vermeer BiDi HoloDesk

LIghtSpace BiDi HoloDesk Vermeer 3D interaction No need for glasses, gloves, HMD Rich free interactions In-Volume interaction Full 360-degree view display