iCub a shared platform for research in robotics & AI Genoa - PowerPoint PPT Presentation

object recognition self-supervised strategies kinematics motion Human Robot Interaction is a new and natural application for visual recognition In robotics settings strong cues are often available, therefore object detectors can be avoided Recognition as tool for complex tasks: grasp, manipulation, affordances, pose S.R. Fanello, C. Ciliberto, L. Natale, G. Metta – Weakly Supervised Strategies for Natural Object Recognition in Robotics, ICRA 2013 C. Ciliberto, S.R. Fanello, M. Santoro, L. Natale, G. Metta, L. Rosasco - On the Impact of Learning Hierarchical Representations for Visual Recognition in Robotics, IROS 2013 6/25/2015 53

6/25/2015 54

dataset 6/25/2015 55

iCubWorld dataset (2.0) • Growing dataset collecting images from a real robotic setting • Provide the community with a tool for benchmarking visual recognition systems in robotics • 28 Objects, 7 categories, 4 sessions of acquisition (four different days ) • 11Hz acquisition frequency. • ~50K Images http://www.iit.it/en/projects/data-sets.html 6/25/2015 56

methods 6/25/2015 57

methods Sparse Coding [yang et al. ’09] Sparse Coding [yang et al. ’09] Learned on iCubWorld Learned on iCubWorld Overfeat [sermanet et al. ’14] Overfeat [sermanet et al. ’14] Learned on Imagenet Learned on Imagenet 6/25/2015 58

some questions • Scalability. How do iCub recognition capability decrease as we add more objects to distinguish? • Can we use assumptions on physical continuity to make recognition more stable? • Incremental Learning. How does learning during multiple sessions affect the system recognition skills? • Generalization. How well does the system recognize objects “seen” under different settings? 6/25/2015 59

first results We started by addressing ins instance ance recognition cognition 6/25/2015 60

performance wrt # of objects 6/25/2015 61

exploiting continuity in time 6/25/2015 62

incremental learning Cumulative learning on the 4 days of acquisition. Tested on: accuracy (avg. over classes) • Presen t: test on current day 1 • Causal : test on current and 0.95 0.9 past days 0.85 • Future : test on future days 0.8 (current not included) OF present 0.75 OF causal • Independent : train & test on OF future 0.7 OF Independent current day only 0.65 1 2 3 4 day 6/25/2015 63

what about a week? accuracy (avg. over classes) 1 ? 0.95 0.9 0.85 0.8 OF present 0.75 OF causal OF future 0.7 OF Independent 0.65 1 2 3 4 5 6 7 day 6/25/2015 64

… or a month? accuracy (avg. over classes) 1 0.95 ?? 0.9 0.85 0.8 OF present 0.75 OF causal OF future 0.7 OF Independent 0.65 1 6 11 16 21 26 30 day 6/25/2015 65

6/25/2015 66

6/25/2015 67

Experiments on affordances 6/25/2015 68

Experiments on affordances 6/25/2015 69

Experiments on affordances 6/25/2015 70

Experiments on affordances 6/25/2015 71

6/25/2015 72

3D vision for grasping Input Segmentation Disparity 6/25/2015 73

grasping segmentation point cloud from stereo surface extraction minimum bounding box grasp selection from experience grasp points from curvature + bias (task dependent) Gori,I; Pattacini, U; Tikhanoff, V; Metta G; (submitted 2013) Ranking the Good Points: A Comprehensive Method for Humanoid Robots to Grasp Unknown Objects 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2013) 6/25/2015 74

6/25/2015 75

force reconstruction 6/25/2015 76

raw data 6/25/2015 77

GP approximation 6/25/2015 78

6/25/2015 79

From: Graziano 1999 From: Graziano et al. 2006 from : Fogassi L., Gallese V., Fadiga L., Luppino G., Matelli M., Rizzolatti G. Coding of peripersonal space in inferior premotor cortex (area F4) . Journal of Neurophysiology 76 (1) 1996. 6/25/2015 81

spinal reflexes walking behavior: cat rehabilitated to walk after complete spinal cord transection wiping reflex: an irritating stimulus elicits a wiping movement precisely directed at the stimulus location from: Poppele, R., & Bosco, G. (2003). Sophisticated spinal contributions to motor control. Trends in Neurosciences, 26 (5), 269-276. 6/25/2015 82

visuo-tactile fusion 6/25/2015 83

double touch From two fixed-base chain to a single floating-base serial chain  12 dof 6/25/2015 84

6/25/2015 85

6/25/2015 86

receptive fields Receptive field: a cone that extends up to 0.2m and angle of 40  • 6/25/2015 87

single taxel model 6/25/2015 88

no noise 6/25/2015 89

noisy data 6/25/2015 90

with double touch 6/25/2015 91

visual tracker 6/25/2015 92

external stimulation 6/25/2015 93

6/25/2015 94

extending peripersonal space 6/25/2015 95

6/25/2015 96

6/25/2015 97

6/25/2015 98

what future? iCub new tech iCub3.0 iCub2.0 now the future? 6/25/2015 99

6/25/2015 100

6/25/2015 101

“How old are you?” she wanted to know. “Thirty-two,” I said. “Then you don’t remember a world without robots. There was a time when humanity faced the universe alone and without a friend. Now he has creatures to help him; stronger creatures than himself, more faithful, more useful, and absolutely devoted to him. Mankind is no longer alone. Have you ever thought of it that way?” 6/25/2015 102