Yeonchool Yeonchool Yeonchool PARK & Omar AIT Yeonchool - - PowerPoint PPT Presentation

Yeonchool Yeonchool Yeonchool Yeonchool Yeonchool Yeonchool Yeonchool Yeonchool PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT-

• AIDER

AIDER AIDER AIDER AIDER AIDER AIDER AIDER Institut Institut Institut Institut Institut Institut Institut Institut Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS

ARMS project

RGB-D camera Robot Effector Cutting path

1/ Registration

Current view of meat part Model Local surface mesh from current view Recognition + Registration

2/ Tracking surface deformation during muscle separation

• High variability in meat parts :
• Appearance (color, texture, specularity,…)
• Shape (many bovin species, approximative cutting,…)
• Size
• Robust to partial occlusion
• Robust to partial occlusion
• Must work in real time
• Accuracy in pose computation

• RGB-D image acquisition
• Scene segmentation
• Meat surface mesh generation
• Meat surface mesh generation
• Scene-to-model registration

• Calibrated procam system

RGB caméra + structured light (Currently Kinect)

2D Images 3D Points cloud

(Currently Kinect)

• 3D Point Cloud with associated RGB data
• Data are expressed in camera coordinate

system

3D Points cloud

• Select only data corresponding to the

meat part among the hole point cloud meat part among the hole point cloud

• Use both 3D and 2D information

3D 3D 3D 3D Segmentation Segmentation Segmentation Segmentation

2D Images

Boundary: border between 3D point and background. Basing on color histograms

• Obtained contours are closed
• RGB-D data corresponding to

Semistep: how far from each

• ther between points

Fold : surface discontinuity (normal difference)

• RGB-D data corresponding to

meat part are selected

Transform brut data (point cloud) into a continous surface

Reconstructed Mesh Calculation of Normal from Segmented Patch

Sphere generation to create auxiliary points

Triangulation and cleaning

• Establish point correspondences
• Compute the best rigid

transformation between the

[ R , T ] Model mesh Current view mesh

transformation between the current mesh nd the model

• Existing approaches :

RANSAC (Fischler and Bolles, 1981), ICP (Besl & McKay, 1992)

[ R , T ] Model mesh Current view mesh

• Huge number of combinations
• Risk of high outlier ratio due to

variability of models

• Johnson & Hebert (1999)
• Coordinate transform
• Object centered representation

Andrew E. Johnson and Martial Hebert, 06

• Object centered representation

• Each model is a polygonal mesh M with vertices
• One spin image for each vertex

Matching score improvement

• Bhattacharyya Distance
• Stochastic distance
• Mutual correspondence consistency checking

[ , , , , , ]T z x y z α β γ = = = =

• Compute the set 6 parameter (translation and rotation)

2 1

( ) ( )

b

N i i i

E z a Rb T

= = = =

= − + = − + = − + = − +

∑ ∑ ∑ ∑

• Which minimizes the cost function (least square optimization)
• Eliminate outliers using a RANSAC algorithm

• An initial path is given by the

model

• As the robot effector proceeds to

muscle separation, the cutting « valley » deformations are tracked in real time

• Non-rigid ICP based tracking :
• Iteratively searches the nearest neighbours

whith the highest curvature

• Calculates affine transform using Non-rigid ICP
• Calculates affine transform using Non-rigid ICP

(Minimize deformation energy, Minimize alignment error, Maximize regions of overlap)

• Transform the guide-path points
• Guide-path refinement using polynomial

fitting to generate smooth curve

• Library model :

1000 Points, built using sfm methods from a classical camera image sequence

• Tests with 3 different meat parts
• RGB-D camera :

Kinect with resolution 640 × 480, 32 bits, 30 fps

Performances :

• Registration : 3 to 4 seconds with a model of 1000 points
• Tracking : 30 fps (Kinect frame rate)
• Qualitative evaluation due the difficulty to generate large data sets

with real meat and groundtruth values

• An active vision system enabling free form surface registration and tracking

for boine muscle separation

• Registration processing time is sufficient for our application
• Tracking frame rate must be improved to enable visual servoing

Tracking frame rate must be improved to enable visual servoing

• Need for evaluation with large model data bases
• Take into account the mechanical deformation model to improve the tracking

efficiency and robustness