Model-free Approach to Garments Unfolding Based on Detection of - - PowerPoint PPT Presentation

Model-free Approach to Garments Unfolding Based on Detection of Folded Layers

Jan Stria, Vladim´ ır Petr´ ık, V´ aclav Hlav´ aˇ c

Czech Institute of Informatics, Robotics and Cybernetics Czech Technical University in Prague Czech Republic

26 September 2017

1 / 14

Project CloPeMa (Clothes Perception and Manipulation)

◮ Funded by European Commission in FP7, 2012–2015 ◮

Czech Technical University, University of Genoa, CERTH, University of Glasgow

◮ Fully autonomous pipeline for garments classification,

unfolding, flattening and folding using a dual-arm robot

2 / 14

Problem formulation

◮ Input: Unknown piece of garment folded once or more times

with the top folded layers not overlapping

◮ Goal: Repeatedly detect one folded layer, estimate the

folding axis and unfold it with two cooperating robotic arms.

3 / 14

Motivation

◮ Geometric unfolding1

◮ The garment is grasped and partially unfolded in the air. ◮ The remaining fold is detected after placing it on a table. 1Dimitra Triantafyllou et al. “A geometric approach to robotic unfolding of

garments”. In: Robotics and Autonomous Systems (2016)

4 / 14

Related work

◮ Foldable templates1

◮ Input: RGB image ◮ Partial template fitting to

the garment contour

◮ Folding axis generation

◮ Depth segmentation2

◮ Input: Depth map ◮ Watershed segmentation ◮ Checking various

unfolding directions

2David Estevez et al. “Towards Robotic Garment Folding: A Vision

Approach for Fold Detection”. In: Proc. ICARSC (2016)

5 / 14

Input

◮ RGBD sensor ASUS Xtion attached to the wrist is positioned

above the garment and oriented downwards.

◮ Acquire single image and depth map of the garment.

950 990 1030 1070 [mm]

6 / 14

Layers detection

10 20 [mm]

◮ Layers detection is formulated as pixels labeling:

p → zp ∈ {T, B}

◮ Combine information provided by the image and heights:

◮ Pixels from the top (folded) layer are higher above the table

than the pixels from the bottom layer.

◮ The boundary between layers is coincident with image edges. 7 / 14

Energy minimization 1

◮ Pixels labeling formulated as energy minimization problem:

Z ∗ = arg min

Z∈{T,B}|P|

• p∈P

Up(zp) +

• {p,q}∈N

Vp,q(zp, zq)

◮ Unary potential Up evaluates how the pixel height H(p)

corresponds to the estimated mean height of the bottom layer µB and the top layer µT = 2µB: Up(zp) = − log N

• H(p); µzp, σ2

3 6 9 12 15 Top Bottom

8 / 14

Energy minimization 2

◮ Pairwise potential Vp,q depends on the spatial d(p, q) and

visual g(I, p, q) difference of the neighboring pixels: Vp,q(zp, zq) = γ1 + γ2 zp = zq d(p, q) exp

• −g(I, p, q)

2E[g]

• ◮ Solved by finding the minimum cut of a specific graph.

◮ The largest top layer is chosen for unfolding.

9 / 14

Folding axis estimation

◮ The folding axis must form an approximate segment on the

garment contour.

◮ The garment is unfolded virtually over each candidate

folding axis and the best one is selected.

10 / 14

Robotic manipulation

◮ Cooperated manipulation of two robotic arms:

◮ One arm is holding the garment to prevent it from slipping. ◮ The other arm grasps the top layer and unfolds it.

◮ Test various grasping and holding candidates. ◮ The holding gripper follows a triangular unfolding path.

11 / 14

Vision experiments

◮ Dataset containing 13 garments of various types, colors and

materials; each posed in 15 folded configurations Garment Success Failure Layers Axis Jacket 14 / 15 1 Jeans 12 / 15 3 Shorts 14 / 15 1 Skirt (2) 25 / 30 5 Sweater (2) 26 / 30 2 2 Sweatshirt 14 / 15 1 Towel 14 / 15 1 T-shirt (4) 51 / 60 9 Total 87% 11% 2%

12 / 14

Robotic experiments

Garment Success Reason of failure Detection Planning Execution Shorts 3 / 5 1 1 Sweatshirt 4 / 5 1 Towel 5 / 5 T-shirt 5 / 5 Total 17 / 20 2 1

13 / 14

Thank you for your attention. Questions, please? http://bit.do/unfolding Czech Institute of Informatics, Robotics and Cybernetics Czech Technical University in Prague

14 / 14