Manipulation in HRI: How a robot: an Overview Makes physical - - PDF document

3/22/19 1

Manipulation in HRI: an Overview

What is Manipulation?

• How a robot:
• Makes physical changes to the world around it
• Physically interacts with the world and other agents
• Grasping, pushing, carrying,

moving, joining, placing, dropping, throwing, …

• Manipulators
• Arm(s) with end-effectors
• Other types

slide adapted from www.cs.columbia.edu/~allen/F15/NOTES/graspingClass2_2.ppt

Manipulators Manipulators Manipulators Terminology

• UnderactuaFon: Only some joints are directly

controlled

• Compliant/Compliance: SoJ, not rigid, yielding
• Handover: Handing from one agent to another
• IMU: Interial Measurement Unit
• Feed-forward: responds (to some signal) in a pred-

defined way (no feedback incorporated)

3/22/19 2

• Physically alter the world through contact
• As a primary goal
• Not its posi(on
• When is this desirable?
• Dangerous workspaces
• Human-intractable workspaces
• Boring, repeFFve, unpleasant work
• And in HRI?

Manipulation

Tasks Challenges

• Retrieving objects
• Carrying objects
• Placing objects
• Handoff
• Physical assistance
• Carrying, transfer,

feeding, …

• Compliant objects
• Compliant grippers
• TelemanipulaFon
• Scaled manipulaFon
• Human manipulaFon
• CollaboraFon

Manipulation in HRI Grasps

• Grasp:
• A set of contact points on an object’s surface
• Goal: constrain object’s movement

www www.intechopen.com .intechopen.com/books/r /books/robot-ar

• bot-arms/r

ms/robotic-grasping-of-unknown-objects1

• botic-grasping-of-unknown-objects1

news.nationalgeographic.com news.nationalgeographic.com/news news/2009/05/090505-r /2009/05/090505-robot-hand-pictur

• bot-hand-picture.html

e.html

Grasps

• Grasps vary by:
• Hand (gripper)
• Object being grasped
• Topology, topography,

mass, surface, …

• Type of moFon desired
• For each hand or

hand/object pair:

• Where to grasp it?
• How hard?
• Then what?
• AddiFonal constraints (e.g., don’t spill)

www www.madry .madry.pr .pro León, Morales, Sancho- León, Morales, Sancho-Bru

• Bru. Robot Grasping Foundations. 2013

. Robot Grasping Foundations. 2013

Tool use

• ol use

Drinking Drinking

• Grasps are not obvious (easy to calculate)
• Any given object has arbitrary contact points
• Hand has geometry constraints, etc.
• Synthesized trial-and-error
• For a hand/object pair:
• Different grasp types planned and analyzed
• Real trial and error

The Grasping Problem

www www.cs.columbia.edu/~cmatei/ .cs.columbia.edu/~cmatei/graspit graspit/ / www www.pr .programmingvision.com/r

• grammingvision.com/resear

esearch.html ch.html www www.cc.gatech.edu .cc.gatech.edu/gvu gvu/people/faculty/ /people/faculty/nancy nancy.pollar .pollard/grasp.html grasp.html

• Grasp synthesis: Find suitable set
• f contacts, given:
• Object model
• Constraints on allowable contacts
• Grasp points are determined
• Mostly assume point contacts
• Larger areas usually discreFzed
• Contact model defines the force the

manipulator exerts on contact areas

• Grasp analysis
• Is that grasp stable?

Grasp Planning

León, Morales, Sancho- León, Morales, Sancho-Bru

• Bru. Robot Grasping Foundations. 2013.

. Robot Grasping Foundations. 2013. www www.intechopen.com .intechopen.com/books/r /books/robot-ar

• bot-arms/r

ms/robotic-grasping-of-unknown-objects1

• botic-grasping-of-unknown-objects1

3/22/19 3 Robocup@Home

www.robocupathome.org

• Takes place in simulated kitchen & living room
• Examples of past tasks:
• Speech and Person Recogni(on: idenFfy unknown people and answer quesFons

about them and the environment

• Cocktail Party: learn and recognize previously unknown people, fetch orders
• Help-me-carry: Help bring groceries into the home from outside
• Storing Groceries: Storing new groceries in the cupboard next to objects of the

same kind that are already there

• Dishwasher Challenge: remove all dishes from a table and put in dishwasher
• Restaurant: two robots move within and environment to handle human requests,

such as delivering drinks or snacks, while people are walking around

• ∃ several different “RoboCup” compeFFons/challenge areas

Prismatic Impactive Gripping

https://www.youtube.com/watch?v=qKZLx1wtFCk

Soft Impactive Gripping

https://www.youtube.com/watch?v=qPVt0bZtNAM

“…relies on two kinds of soft robot technology: pneumatics and dielectric elastomer actuators.” [Science Magazine, Jan. 2018]

Soft Pneumatic Impactive Gripping

https://www.youtube.com/watch?v=gI0tzsO8xwc

And Then There’s This

youtu.be/0d4f8fEysf8

Reactive Gripping

• React to sensaFon of touching something
• Rotate to grasp
• High-fricFon, compliant fingers
• Minimal sensor suite: IMUs only (wow!)
• Works with human-shaped hands only
• ArFfact of data collecFon
• Don’t consider objects from above
• Tested detecFon of objects from above by hi`ng it with a

wrench; 88% of the Fme it read as a whack from above ~86% success

• n tennis ball

3/22/19 4 Example Reactive Grasp Data and Applications

• Data gathering: IMU glove, wrist locaFon via

markers, hapFc feedback

• First contact only with distal phalanges (fingerFps) –

is that realisFc?

• 15 seconds hold??
• No feedback for

unexpected events

• Not usually a

good HRI assumpFon

Simplifications

• Reject baseline wrist pose
• Assumes no “jerk” in movement
• Measure wrist 3d rotaFon, but the controller only

allows roll

• How is this reacFve?
• Doesn’t handle addiFonal constraints (torque,

direcFon, weight)

• Is this really handover?

Reactive Primitives for Soft Hands

https://www.youtube.com/watch?v=N03WTBK5eMw

Multimodal Shared Autonomy

• Try to use eye gaze to predict human intent
• SubstanFal data-gathering experiments
• Eyes, pupillary response
• JoysFck control
• Heavy quality filtering
• Of course, can’t filter
• ut your actual users

Pupillary Response

• Pupils suggest joysFck is

more (mental) effort

• Variance high
• Affected by blinking,

brief unrelated saccades, peripheral vision

• Gaze tracking is super

hard

3/22/19 5 Eye Movements

• “Planning glances”: Not moving joysFck
• “Monitoring glances”:

Checking status

• Mostly while moving

the arm, not rotaFng

• Pakerns about what

kinds of gaze are associated with what acFons

Conclusions

• These are noisy signals!
• There is useful informaFon in gaze
• Gaze and control are not randomly distributed
• Can take advantage of those pakerns
• Robot behavior has an effect on all modaliFes

Discussion

• What is the problem addressed?
• What is the approach they take?
• Did it work?
• Did they choose good experiments / metrics?
• What are their conclusions?
• What do we think of this paper?