ARMS project Yeonchool Yeonchool Yeonchool PARK & Omar AIT Yeonchool Yeonchool Yeonchool Yeonchool Yeonchool PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT PARK & Omar AIT- - -AIDER - - - - - AIDER AIDER AIDER AIDER AIDER AIDER AIDER Institut Institut Institut Institut Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS Institut Institut Institut Institut Pascal, CNRS Pascal, CNRS Pascal, CNRS Pascal, CNRS
RGB-D camera 1/ Registration Robot Effector Cutting path Local surface mesh from current view Current view of meat part Recognition + Registration Model 2/ Tracking surface deformation during muscle separation
• High variability in meat parts : � Appearance (color, texture, specularity,…) � Shape (many bovin species, approximative cutting,…) � Size • Robust to partial occlusion • Robust to partial occlusion • Must work in real time • Accuracy in pose computation
• RGB-D image acquisition • Scene segmentation • Meat surface mesh generation • Meat surface mesh generation • Scene-to-model registration
• Calibrated procam system RGB caméra + structured light (Currently Kinect) (Currently Kinect) 2D Images • 3D Point Cloud with associated RGB data • Data are expressed in camera coordinate system 3D Points cloud
3D Points cloud • Select only data corresponding to the meat part among the hole point cloud meat part among the hole point cloud 3D 3D 3D 3D Segmentation Segmentation Segmentation Segmentation • Use both 3D and 2D information 2D Images
Boundary: border between 3D point and background. Basing on color histograms • Obtained contours are closed • RGB-D data corresponding to • RGB-D data corresponding to meat part are selected Fold : surface discontinuity (normal difference) Semistep: how far from each other between points
Transform brut data (point cloud) into a continous surface Calculation of Normal from Segmented Patch Sphere generation to create auxiliary points Triangulation and cleaning Reconstructed Mesh
• Establish point correspondences [ R , T ] • Compute the best rigid transformation between the transformation between the current mesh nd the model Current view mesh Model mesh
• Existing approaches : RANSAC ( Fischler and Bolles, 1981 ), [ R , T ] ICP ( Besl & McKay, 1992 ) • Huge number of combinations • Risk of high outlier ratio due to variability of models Current view mesh Model mesh
• Johnson & Hebert (1999) • Coordinate transform • Object centered representation • Object centered representation Andrew E. Johnson and Martial Hebert, 06
• Each model is a polygonal mesh M with vertices • One spin image for each vertex
Matching score improvement • Bhattacharyya Distance • Stochastic distance • Mutual correspondence consistency checking
• Compute the set 6 parameter (translation and rotation) [ , , , , , ] T = = = = x y z α β γ z • Which minimizes the cost function (least square optimization) N b ∑ ∑ ∑ ∑ 2 ( ) ( ) = = = = − − − − + + + + E z a Rb T i i 1 i = = = = • Eliminate outliers using a RANSAC algorithm
• An initial path is given by the model • As the robot effector proceeds to muscle separation, the cutting « valley » deformations are tracked in real time
• Non-rigid ICP based tracking : - Iteratively searches the nearest neighbours whith the highest curvature - Calculates affine transform using Non-rigid ICP - Calculates affine transform using Non-rigid ICP ( Minimize deformation energy, Minimize alignment error, Maximize regions of overlap) - Transform the guide-path points • Guide-path refinement using polynomial fitting to generate smooth curve
• Library model : 1000 Points, built using sfm methods from a classical camera image sequence • Tests with 3 different meat parts • RGB-D camera : Kinect with resolution 640 × 480, 32 bits, 30 fps
Performances : • Registration : 3 to 4 seconds with a model of 1000 points • Tracking : 30 fps (Kinect frame rate) • Qualitative evaluation due the difficulty to generate large data sets with real meat and groundtruth values
• An active vision system enabling free form surface registration and tracking for boine muscle separation • Registration processing time is sufficient for our application • Tracking frame rate must be improved to enable visual servoing Tracking frame rate must be improved to enable visual servoing • Need for evaluation with large model data bases • Take into account the mechanical deformation model to improve the tracking efficiency and robustness
Recommend
More recommend
