Incremental learning of motion primitives for full body motions - PowerPoint PPT Presentation

Incremental learning of motion primitives for full body motions Dana Kuli´ c Department of Electrical and Computer Engineering, University of Waterloo, Canada Incremental learning of motion primitives for full body motions – p. 1/43

Learning from Observation for Humanoids Learn to accomplish tasks by observing a human teacher, rather than programming or trajectory planning Take advantage of similar structure between human and robot Suitable for non-expert demonstrators Incremental learning of motion primitives for full body motions – p. 2/43

Related Work [Calinon and Billard 2007] HOAP at EPFL [Ikeuchi et al. 2004] HRP-2 at AIST Incremental learning of motion primitives for full body motions – p. 3/43

Limitations of the current approaches Motions are specified manually by the designer In learning systems, motions are segmented and clustered a-priori Off-line, one-shot training No further learning during the execution stage Incremental learning of motion primitives for full body motions – p. 4/43

Desired System - A robot that cohabits with humans, and learns incrementally over a lifetime of observations - A robot that accumulates knowledge and improves performance over time - Fully autonomous, on-line, continuous learning System Requirements: Autonomous Motion Segmentation Autonomous, On-line Motion Clustering Autonomous Knowledge Management with fast Retrieval Incremental learning of motion primitives for full body motions – p. 5/43

Talk Outline Robot Learning from Observation Representing full-body Motion On-line Segmentation On-line Clustering and Organization Combining Segmentation and Clustering Learning the sequencing of motion primitives Incremental Memory Consolidation Conclusions and Directions for Future Work Incremental learning of motion primitives for full body motions – p. 6/43

Learning from Observation - Mirror Neurons The same neural structure is used for both recognition and generation [Rizzolatti et al. 2001] Incremental learning of motion primitives for full body motions – p. 7/43

Motion Representation by Hidden Markov Models [Inamura et al. 2004] Stochastic model capturing both spatial and temporal variability Model training (learning) is implemented with the Baum-Welch Algorithm Once the model is trained, the same model can be used for both Recognition (Forward Procedure) Generation (either stochastic or deterministic) Factorial HMMs also used for representing motions with greater accuracy [Kuli´ c et al. IROS 2007] Incremental learning of motion primitives for full body motions – p. 8/43

On-line Segmentation Want to segment with no a-priori knowledge of the motions Must make some assumption about the structure of the data Mean velocity falls below a certain value [Pomplun and Matari´ c, 2000] Zero velocity crossing in some dimensions [Fod et al., 2002] Minimize variance [Koenig and Matari´ c, 2006] Same motion will belong to same underlying distribution [Kohlmorgen and Lemm, 2001] [Janus and Nakamura, 2005] Incremental learning of motion primitives for full body motions – p. 9/43

Stochastic Segmentation [Kohlmorgen and Lemm, 2001] Embed the data into a higher-dimensional space x t = ( � y t − ( m − 1) τ ) � y t , � y t − 1 , . . . , � Estimate the density distribution over a sliding window of length W W − 1 x t − w ) 2 p t ( x ) = 1 (2 πσ 2 ) d/ 2 exp ( − ( x − 1 � � ) 2 σ 2 W w =0 Incremental learning of motion primitives for full body motions – p. 10/43

Computing the distance between states Can compute the distance between windows based on integrated square error between two pdfs � ( p t 1 ( x ) − p t 2 ( x )) 2 dx d ( p t 1 , p t 2 ) = 1 d ( p t 1 ( x ) , p t 2 ( x )) = W 2 (4 πσ 2 ) d/ 2 W − 1 x t 1 − v ) 2 [ exp ( − ( x t 1 − w − � � � ) 4 σ 2 w,v =0 x t 2 − v ) 2 − 2 exp ( − ( x t 1 − w − � � ) 4 σ 2 x t 2 − v ) 2 + exp ( − ( x t 2 − w − � � ) (1) 4 σ 2 Incremental learning of motion primitives for full body motions – p. 11/43

Defining a stochastic model over the data Define an HMM over a set of sliding windows. Observation Function: 2 πς exp ( − d ( p s ( x ) , p t ( x )) 1 √ p ( p t ( x ) | s ) = ) 2 ς 2 State Transition Model:  k if i = j ;  k + N − 1 a ij = 1 if i � = j .  k + N − 1 Incremental learning of motion primitives for full body motions – p. 12/43

Segmentation via the Viterbi Algorithm Find the optimum state sequence for the specified HMM, given the actual data sequence A simple example: t Incremental learning of motion primitives for full body motions – p. 13/43

Segmentation via the Viterbi Algorithm Find the optimum state sequence for the specified HMM, given the actual data sequence A simple example: t s1 s2 s3 s4 s5 s6 Incremental learning of motion primitives for full body motions – p. 14/43

Segmentation via the Viterbi Algorithm Find the optimum state sequence for the specified HMM, given the actual data sequence A simple example: t s1 s2 s3 s4 s5 s6 Optimum State 1 1 1 4 4 4 Sequence (Viterbi) Incremental learning of motion primitives for full body motions – p. 15/43

Improving the Segmentation [Kuli´ c Nakamura IROS 2008] Bias state transition model towards known states k  if i = j ; C    1 a ij = if i � = j and i ∈ S t ; C  K s  if i � = j and i ∈ S p .  C Modify pdf based on active joints in the known state 1 D w ( p t 1 ( x ) , p t 2 ( x )) = L 2 (4 πσ 2 k ) d/ 2 L − 1 x t 1 − j ) 2 [ exp ( − W ( x t 1 − i − � � � ) 4 σ 2 k i,j =0 x t 2 − j ) 2 − 2 exp ( − W ( x t 1 − i − � � ) 4 σ 2 k x t 2 − j ) 2 + exp ( − W ( x t 2 − i − � � ) (2) 4 σ 2 k Incremental learning of motion primitives for full body motions – p. 16/43

On-line clustering and hierarchy formation [Kuli´ c et al. ISRR 2007, IJRR 2008] Use HMM representation to abstract motion patterns as they are perceived Cluster individual motion patterns incrementally, based on intra-model distances Use formed clusters to form group models Autonomously select appropriate model type, based on model distances in the considered region of the motion space 2 2 2 2 18 18 13 13 13 13 1 1 1 1 4 4 4 4 15 15 12 12 12 12 16 8 8 6 6 6 6 10 10 10 10 20 14 3 9 14 14 3 3 3 9 11 7 7 11 11 7 5 5 5 5 17 17 19 19 Incremental learning of motion primitives for full body motions – p. 17/43

Algorithm Pseudo-Code Following observation of each motion sequence: Step1 Encode observation sequence O i into an HMM λ i Step2 Calculate the distance between λ i and each existing behavior group model λ Gj Step3 Place λ i into the closest group G c Step4 Cluster all exemplars within G c Step5 If a sub-group forms, form a new node G n , containing the exemplars of the cluster Step6 Using the observation sequences from the exemplars in G n , form the new sub-group model λ Gn Incremental learning of motion primitives for full body motions – p. 18/43

Combining segmentation and Clustering [Kuli´ c et al. ICRA 2008] x y θ z θ t Incremental learning of motion primitives for full body motions – p. 19/43

Experiments x y θ z θ t 4 minutes of continuous whole body motion data of a single subject from motion capture data data is converted to a 20DoF humanoid model by online inverse kinematics First, test the basic segmentation algorithm, with no known states, and compare with manual segmentation Incremental learning of motion primitives for full body motions – p. 20/43

Testing the Segmentation x y θ z θ t Next, test the improvements obtained through adding known motions Provide manually extracted primitives as exemplars Incremental learning of motion primitives for full body motions – p. 21/43

Segmentation Results WMID LAR LAL SL SR RKE RKR WLR WRL RAR RAL Current Motion Manual Segmentation Segmentation Result (with known motions) Segmentation Result (no known motions) 70 72 74 76 78 80 82 84 Time [seconds] Algorithm Correct False Pos False Neg Basic 128 65 43 Scaffolded (with Squat and Kick) 139 59 32 Worst performance occurs at switching points where few joints are moving Sample Video Incremental learning of motion primitives for full body motions – p. 22/43

Testing the Combined Segmentation and Clustering x y θ z θ t Present the complete 4min sequence and apply segmentation The leaf nodes of the resulting tree are used to scaffold the segmentation To facilitate analysis, 4min sequence is presented repeatedly (epochs), and new exemplars are added to the segmentation module at the end of each epoch Incremental learning of motion primitives for full body motions – p. 23/43

After Epoch 1 Epoch = 0 WRS LAL / WLS partial RAR SL WF LAL/partial RAR SL WRS WLS WF Leaf Groups Formed Example Extracted Motion: Right Arm Raise Incremental learning of motion primitives for full body motions – p. 24/43

After Epoch 2 Epoch = 1 WRS WLS LAL RAR SL WF KR LAL RAR SL KR WRS WLS WF Leaf Groups Formed Example Extracted Motion: Left Arm Lower Incremental learning of motion primitives for full body motions – p. 25/43