Large Shared Haptic Virtual Environments Nakul Chaudhari - - PowerPoint PPT Presentation

Large Shared Haptic Virtual Environments

Master Thesis Presentation

Nakul Chaudhari nakul.chaudhari@tum.de Supervisors: Clemens Schuwerk clemens.schuwerk@tum.de Marc-Oliver Pahl pahl@net.in.tum.de

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Agenda

• Introduction

□ haptic device and a computer haptic application □ motivation

• What is a networked haptic application?

□ traditional architectures for collaborative haptic applications

• Scaling haptic virtual environments
• Area of interest

□ static partitioning □ dynamic partitioning

• Results

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Simple haptic device

with 3 degrees of freedom

resting actor using the device

point, HIP

movement along various DOF - video

HIP - haptic interface pointer position

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Simple haptic device

with 3 degrees of freedom

resting actor using the device

point, HIP

movement along various DOF - video

HIP - haptic interface pointer position

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

4

Computer haptics - graphical rendering video surface effects - 1-smooth 2-water 3-rough 4-magnetic

1 2 3 4

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

4

Computer haptics - graphical rendering video surface effects - 1-smooth 2-water 3-rough 4-magnetic

1 2 3 4

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

4

Computer haptics - graphical rendering video surface effects - 1-smooth 2-water 3-rough 4-magnetic

1 2 3 4

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Motivation

• Previous work in haptics generally surrounded spatially small worlds
• This thesis: different users collaborating with other users on different

tasks

• Needs: architecture which supports such scaling

5

scale world and collaborating users maintain consistency, transparency, and realistic feedback

• ne single and big VE

collaborate on different tasks translate camera, zoom in / out

• nly one task

no change in camera state

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Abbreviations

• HIP - haptic interface pointer

□ position of the haptic device (or the user)

• VE - virtual environment or virtual world
• P2P - peer to peer
• RTT - round trip time in network communication
• AOI - area of interest

□ area surrounding a user in the VE, e.g. volume covered by a sphere drawn around the user

6 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Networked haptic applications & Traditional architectures

7 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Networked haptic applications

8

Operator Visual Display Haptic Display Local Control Loop Local Control Loop Teleoperator Network Sensors and Actuators p

Example scenario - teleoperation

adopted from Steinbach et al., Haptic data compression and communication. IEEE Signal Processing Magazine

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Networked haptic applications

8

Operator Visual Display Haptic Display Local Control Loop Local Control Loop Teleoperator Network Sensors and Actuators p

Example scenario - teleoperation

adopted from Steinbach et al., Haptic data compression and communication. IEEE Signal Processing Magazine

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Networked haptic applications

9

Network Device position Loop 1kHz *P .D. Force, Audio, Video

*P .D. Perceptual deadband scheme for packet reduction Separate thread at 30 Hz

• Challenges

□ high header to payload ratio □ high update rate of 1kHz □ stability issues due to network delay

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

10

Position Sensing Collision Detection Response Force Computation Virtual Object Database Force Torque Network / Storage Haptic Feedback (1000 Hz) Human-System Interface Communication Visual Feedback (30-100 Hz) Visual Rendering Physics Engine

!"# !$#

Computer haptics

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Networked haptic applications & Traditional architectures for collaboration

11 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

12

collaboration - video

Collaboration between networked Haptic Applications

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

12

collaboration - video

Collaboration between networked Haptic Applications

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Simple client server

• Issues

□ control loop □ RTT time required □ not scalable

• Features

□ consistency □ less resources at client needed

13

!"#$"# %&"#'( %&"#') %&"#'* +",-.#/

0.&1,1.2'.#'$"3.41,5'.6'789 :;0,14&<'$1=".<';>=1.

Graphics Engine Physics Engine Force Rendering

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Consistency client server (CoS)

14

• Improvement

□ local force rendering, hence stable

• Issues

□ RTT time required □ not scalable

• Features

□ consistency

Physics Engine !"#$%&'#()*+',-', .%',*/

0$&1*("2)*"3*45

.%',*6

0$&1*("2)*"3*45

.%',*7

0$&1*("2)*"3*45

8'&0",9

2"%$&$"#*",*-':"($&)*"3*;<=

• $,&>?:*"@A'(&*>2B?&'%

Graphics Engine Force Rendering

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Peer to peer (P2P)

15

!"#$%&

'()*%+,-.%,/%01

!"#$%2

'()*%+,-.%,/%01

!"#$%3

'()*%+,-.%,/%01

4#)',$5

6(7($#+)(,89:%*9-)(+% +,;;<8(+9)(,8%6#)'##8%<"#$"

Graphics Engine Physics Engine Force Rendering

• Improvement

□ no RTT required □ more scalable then previous approaches

• Issues

□ inconsistent virtual world: as every user is using his current HIP and other users HIP state which is ‘delay’ time old

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Inconsistency

16

user 1 user 2

Naive p2p implementation

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Scaling virtual world

17 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Scaling virtual environment

• Virtual world containing simple cubes
• Modules of update haptics routine

□ which are computed at the rate of 1kHz □ whose computation time required increases linearly with number of

• bjects

18

2 4 6 8 10 12 14 10 20 30 40 160 640 1ms Time required in ms per module Number of objects

1ms - max limit on iteration time due to 1kHz update rate ConvertToGlobal detectCollision updateVEState Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Area of Interest

19 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Area of interest (AOI) of a user

20

A B Users Virtual objects AOI of a user

Top view of a VE

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

21

Rendering area of interest (AOI) of a user

user moving slightly to the right grid of cubes user/ HIP

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Static partitioning

22

fully meshed P2P, considering user A AOI of user A and B

A B Figure 7.2: 1.5 A B Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Static partitioning

23

distant inconsistency - add a server, who can be queried for latest VE state A and B’s AOI intersect, exchange HIP only between intersecting users

A B A B Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Static partitioning

24

add multiple servers to reduce bottleneck local inconsistency - change these servers to consistency servers

1 2 5 6 3 4 Non-particpating zone servers Static zones

assign every server a particular volume of the VE

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Zones

• Zone is a specific volume in the VE
• Every Zone has

□ one consistency server □ dedicated server or □ a collaborating user itself □ participating users □ virtual objects

• only consistency server has physics engine

25 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Static partitioning of the virtual environment

• Zones - fixed server, shape, size and location
• issue - handover of objects and user from one zone to another

26

1 2 5 6 3 4

border separating 2 neighboring zones Object crossing zone (Owned by user 1) stiff Object penetrating into other user Object owned by user2

1 1 1 2

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

27

2nd proposed scheme Dynamic partitioning Differ by the way zones are formed and maintained

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Dynamic partitioning

28

□ How are zones/groups formed? □ What is a master user?

A B C D E F G H I J K

Objects of same color are maintained by the master of same color who is a user in the same group

A B C D E F G H I J K Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

29

Dynamic partitioning - CoS groups

Objects of same color are maintained by the master of same color who is a user in the same group

A B C D E F G H I J K

Consistency server groups

A B C D E F G H I J K Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

30

Dynamic partitioning - inter group

A B C D E F 4 F G H I J K 4 E,F 4 E,F F E Collect latest position

• f the user

E C Send latest state

• f members and objects

C A Relay received Info

□ Why is inter group communication necessary? □ Carried out at a medium and lower frequency

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

31

Dynamic partitioning - intra group

□ Which information does the master user collect? □ Which information does he send to which user? □ Carried out at a higher frequency

A B C 1 2 3 3 A User Master Collect latest position

• f the user, here

position of A Master User Send latest state

• f the objects, which

are only in his AOI, here state of object 3

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Comparison

32

Static Partitioning Dynamic Partitioning

dedicated servers needed not needed, as users themselves take the responsibility of consistency servers multicasting is possible as we always have the same servers maintaining a zone not possible, due to constant changing of masters handover of users and objects is an issue dynamically expanding AOI automatically accommodates interacting users causes inconsistency when a server become unavailable if a server wishes to leave, we can elect another user as master

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

33

dynamically expanding AOI to form group - video user1 screen, user1 is near row1 user2 is near 2nd row from bottom

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

33

dynamically expanding AOI to form group - video user1 screen, user1 is near row1 user2 is near 2nd row from bottom

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Packet rate reduction

• Dead Reckoning

□ is the process of estimating state of different entities in VE to reduce the packet rate □ in our case performed at the consistency server □ the packets are only triggered when the estimation differs by more than a threshold

• Forward channel

□ HIP position of user - position, linear velocity

• Backward channel

□ object state - position, orientation, linear velocity and angular velocity

• Prediction - in between updates, receiver can use last known values

for prediction

34 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Results

Interaction between a master and member user

35 a b c d 1 e f g h 2 a b c d 1 e f g h 2

Initially both users are placed far away from each

• ther

User2 now moves into users1’s AOI zoomed part of the VE

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

forward channel for user1 (AOI)

36

three levels of frequency exchange Level I - 1000 (Intra group) Level II -10 (Inter group) Level III- 1 (Inter group) Levels changes I-> II -> III

• r

III->II->I

a b c d 1 e f g h 2 a b c d 1 2

a b c d 1 2 100 200 300 400 500 600 700 800 900 1000

• 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Number of HIP packets sent [1/s] time [s]

send with high frequency when new in VE

Inter group

periodic updates

Intra group

# HIP packets time [s]

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Dead reckoning

37

5 10 15 20 25 30 35

• 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Number of HIP packets sent [1/s] time [s]

haptic device is actually moving, reduction due to packet rate haptic device is stationary, prediction will not trigger any updates for such a case # HIP packets time [s]

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

backward channel user1 (AOI)

38

1 2 3 4

• 2000

2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 Number of Object state pakcets sent [1/ms] time [ms]

send latest state of all objects in AOI send latest of objects in AOI, the user might interact with in future #

• bject

state packets time [msec]

a b c d 1

2

a b c d 1 2

Wednesday, April 3, 2013

a b c d 1 2

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Dead reckoning

39

1 2 3 4

• 2000

2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 Number of Object state packets sent [1/ms] time [ms]

1) to other groups updates of only the object who is not stationary 2) to group members 3) leaving group Object state updates are periodically sent when?

• 1 - Periodically to other

groups 2 - Periodically to own group members 3 - when group member is leaving group 4 - Once to users not in group, but might interact with an object in group soon 4)might interact with in future #

• bject

state packets time [msec]

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Effect of constant and variable delay

• Change in master of group

□ no master between the period: old master transfers role to new master

□ operate old master for time equal to delay

□ variable delay

□ operate old master for a considerable long time □ ignore messages from old master when the new master is

• nline
• In general makes it hard for dynamic group maintenance

40 Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Conclusion

• Scheme drastically reduces communication costs.
• Groups scheme cans similarly also reduce computation costs by processing
• nly those objects which are in our group

41

Summary

• Studied the issues with traditional architectures.
• Hybrid of peer to peer and server centric design.
• Exploited the area of interest (AOI) scheme.
• Different methods / parameters for packet rate reduction for inter

and intra group communication.

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

42

Future work

• AOI used here is spherical (circular), we can attempt other AOI

schemes like hexagonal, square, etc.

• Improve dead reckoning in Intra group communication
• Integrate solutions for countering network impairments like variable

delay

• Test with large number of users and objects

Wednesday, April 3, 2013

Technische Universität München Institute for Media Technology

• Prof. Dr.-Ing. Eckehard Steinbach

Thank you all for listening

• Any Questions?

43 Wednesday, April 3, 2013