stereo vision approaches for human to robot handover
play

Stereo Vision Approaches for Human to Robot Handover Aleksej - PowerPoint PPT Presentation

Apr 01, 2023 •350 likes •1.48k views

MIN Faculty Department of Informatics Stereo Vision Approaches for Human to Robot Handover Aleksej Logacjov University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

  1. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  2. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  3. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  4. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 12 / 44

  5. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  6. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  7. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  8. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  9. Basics: Depth Calculation Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration Image Rec- ◮ F.e. pixel in disparity map: calc. tification distance z = T · f d ◮ Straight forward approach with Stereo Cor- linear complexity respondence Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 14 / 44

  10. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  11. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  12. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  13. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  14. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  15. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  16. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global ◮ For whole image at once ◮ Better in noise/occlusion handling [Olofsson, A. 2010] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  17. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global ◮ For whole image at once ◮ Better in noise/occlusion handling [Olofsson, A. 2010] Focus on local A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  18. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  19. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  20. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  21. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  22. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  23. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) C ( x 1 , y 2 , − 1 ) , C ( x 1 , y 2 , 0 ) , C ( x 1 , y 2 , 1 ) , C ( x 1 , y 2 , 2 ) , C ( x 1 , y 2 , 3 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  24. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) C ( x 1 , y 2 , − 1 ) , C ( x 1 , y 2 , 0 ) , C ( x 1 , y 2 , 1 ) , C ( x 1 , y 2 , 2 ) , C ( x 1 , y 2 , 3 ) ◮ Cost Computation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  25. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Window centered at pixel ◮ Taking neighbors into account A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 18 / 44

  26. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work d ◮ Disparity Space Image (DSI) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 19 / 44

  27. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work C d d Min ◮ Disparity Space ◮ For each pixel Image (DSI) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 19 / 44

  28. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Example window-based cost computation [Olofsson, A. 2010] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 20 / 44

  29. Stereo Correspondence Algorithms Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 3 different approaches to compute the matching cost : ◮ (Sum of) Absolute Intensity Difference C SAD ( x , y , d ) ◮ Deep Learning Approach [Luo, W. et al., 2016] ◮ Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 21 / 44

  30. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  31. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  32. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  33. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: Fast, simple. According to [Scharstein, D. et al., 2002] one of the fastest classical approach. ◮ Cons: Bad accuracy (place 8 of 20 according to [Scharstein, D. et al., 2002]) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 23 / 44

  34. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: Fast, simple. According to [Scharstein, D. et al., 2002] one of the fastest classical approach. ◮ Cons: Bad accuracy (place 8 of 20 according to [Scharstein, D. et al., 2002]) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 23 / 44

  35. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 24 / 44

  36. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] ◮ Put one image patch, centered at pixel (x,y) as input (9x9) ◮ Put an image patch, of size (max_disparity,9) as sec. input ◮ Network computes in one iteration , the cost for all given disparities A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 25 / 44

  37. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] ◮ Put one image patch, centered at pixel (x,y) as input (9x9) ◮ Put an image patch, of size (max_disparity,9) as sec. input ◮ Network computes in one iteration , the cost for all given disparities A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 25 / 44

  38. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Training: ◮ Random image patches from the Kitti dataset ◮ Cross entropy loss for multi class classification (disparities) ◮ 6.5 hours training ◮ Testing/Benchmarking: ◮ On Kitti and Middleburry dataset A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 26 / 44

  39. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Training: ◮ Random image patches from the Kitti dataset ◮ Cross entropy loss for multi class classification (disparities) ◮ 6.5 hours training ◮ Testing/Benchmarking: ◮ On Kitti and Middleburry dataset A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 26 / 44

  40. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, compared to other learning approaches (1sec on NVIDIA Titan-X) ◮ As accurate as other learning approaches ◮ Cons: ◮ No comparison to non learning state-of-the art approaches ◮ After calculation, cost aggregation, smoothing done (time consuming) ◮ No CPU runtime A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 27 / 44

  41. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, compared to other learning approaches (1sec on NVIDIA Titan-X) ◮ As accurate as other learning approaches ◮ Cons: ◮ No comparison to non learning state-of-the art approaches ◮ After calculation, cost aggregation, smoothing done (time consuming) ◮ No CPU runtime A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 27 / 44

  42. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  43. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  44. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  45. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, (0.7 sec on i7 CPU with 2.66 GHz ) ◮ Performs well on higher resolution images (900x750) ◮ Better accuracy than other state-of-the-art approaches ◮ Cons: ◮ Non trivial algorithm ◮ 0.7 sec maybe to slow for human-robot-handover A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 29 / 44

  46. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work iCub using ELAS [Nguyen, P. D., et al. 2018] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 30 / 44

  47. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  48. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  49. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  50. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work We need 3 things: 1. Object detection and tracking (create bounding box ) 2. Cutting out the bounding box 3. Use z = T · f to calc. disparity boundaries d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 32 / 44

  51. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work We need 3 things: 1. Object detection and tracking (create bounding box ) 2. Cutting out the bounding box 3. Use z = T · f to calc. disparity boundaries d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 32 / 44

  52. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  53. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  54. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  55. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  56. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Full view A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  57. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Cut out view Full view A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  58. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Cut out view Full view ◮ Detection runtime: approx 2 sec ◮ x coordinates important for disparity A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  59. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Original image left [Middlebury Dataset] Original image right [Middlebury Dataset] ◮ size: 640x438 A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 35 / 44

  60. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ELAS [Geiger, A. 2012] SAD A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 36 / 44

  61. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ELAS [Geiger, A. 2012] SAD ◮ size: 224x376 (approx 3 times smaller) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 37 / 44

  62. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ SAD: ◮ Big: 0.014 sec ◮ Small: 0.006 sec (2 times faster) ◮ ELAS: ◮ Big: 0.24 sec ◮ Small: 0.06 sec (4 times faster) ◮ Time to cut out: 0.0005 sec A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 38 / 44

  63. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  64. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  65. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  66. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  67. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  68. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  69. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  70. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  71. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  72. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016 http://cvg.ethz.ch/teaching/3dvision/ Dense Correspondence / Stereo Tsukuba dataset http://cat.middlebury.edu/stereo/ Stereo Standard stereo geometry Stereo matching

478 views • 35 slides
1 Basic Stereo Derivations Correspondence It is fundamentally ambiguous, even with stereo

1 Basic Stereo Derivations Correspondence It is fundamentally ambiguous, even with stereo

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images Subproblems: Calibrating camera positions. Finding all corresponding points (hardest part) Computing depth or surfaces. Slide credits

107 views • 7 slides
Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo

Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo

Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo rectification? Reproject image planes onto a common plane parallel to the line between camera centers Recall: Stereo Rectification What can we

597 views • 26 slides
Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images Subproblems: Calibrating camera positions. Finding all corresponding points (hardest part) Computing depth or surfaces. Slide credits

468 views • 42 slides
Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical

Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical

Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical and Computer Engineering University of Toronto Introduction What is Stereo Vision? The ability of finding the depth information encoded

384 views • 26 slides
Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar

Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar

CS 376: Computer Vision - lecture 17 3/26/2018 Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar geometry and the epipolar constraint Case example with parallel optical axes General case with

300 views • 18 slides
Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is

Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is

2/9/16 Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is a robot? u What counts as interaction? u Robot(ic)s, for our purposes, is where computation meets the physical world Principles of u

364 views • 4 slides
Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura,

Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura,

ICRA07 2007 IEEE International Conference on Robotics and Automation Full Day Workshop on Roboethics Rome, April 14th, 2007 Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura, M. L. Sica, A.

729 views • 34 slides
Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1

Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1

29/05/2018 Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1 30/05/2018 Institut Mines-Tlcom IMA 4509 - Stereo vision 3D Visual content Our goal: Estimate 3D world properties from

1.27k views • 122 slides
SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing

SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing

SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing University of Portsmouth students with the best possible sports package. We are exploring a new partnership between Sport & Recreation and the

89 views • 8 slides
Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo

Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo

Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo epipolar lines (x 2 , y 1 ) (x 1 , y 1 ) Two images captured by a purely horizontal translating camera ( rectified stereo pair) x 2 -x 1 = the disparity

650 views • 64 slides
Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro Computer Vision Announcements Midterm Percentage Score Out of Mean 67 52. 78 pts. Median 62 48 High 96 75 Low 24 18 CSE152,

315 views • 17 slides
Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse

Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse

Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse Thomason University of Texas at Austin Ph.D. Defense Human-Robot Dialog 2 Human-Robot Dialog alert me if her heart rate decreases bring

1.73k views • 142 slides
Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches

Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches

Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches Michael Bleyer 1 , Sylvie Chambon 2 , Uta Poppe 1 and Margrit Gelautz 1 1 Vienna University of Technology, Austria 2 Laboratoire Central des Ponts et

880 views • 58 slides
Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI

Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI

Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI Luca Iocchi DIAG, Sapienza University of Rome, Italy Outline Robot motion: main feature characterizing HRI from HCI Human-robot distance

437 views • 21 slides
COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory

COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory

COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory University of Illinois Urbana-Champaign MEET THE ROBOTS WHAT DOES A ROBOT CARE ABOUT WHEN NAVIGATING? Current Location and Destination (possibly

658 views • 30 slides
Computer Vision Levente Hajder, Dmitry Chetverikov Etvs Lornd University, Faculty of

Computer Vision Levente Hajder, Dmitry Chetverikov Etvs Lornd University, Faculty of

Computer Vision Levente Hajder, Dmitry Chetverikov Etvs Lornd University, Faculty of Informatics Hajder, Csetverikov (Faculty of Informatics) 3D Computer Vision 1 / 73 Basics of Stereo Vision Image-based 3D reconstruction 1 Geometry

1k views • 73 slides
1 2 CS 532: 3D Computer Vision Lecture 2 Enrique Dunn edunn@stevens.edu Lieb 310 Image

1 2 CS 532: 3D Computer Vision Lecture 2 Enrique Dunn edunn@stevens.edu Lieb 310 Image

1 2 CS 532: 3D Computer Vision Lecture 2 Enrique Dunn edunn@stevens.edu Lieb 310 Image Formation Based on slides by John Oliensis 3 Lecture Outline Single View Geometry 2D projective transformations Homographies Robust

1.21k views • 92 slides
CSE 152 Section 5 HW2: Stereo Geometry April 29, 2019 Owen Jow Stereo: two views. Why is one

CSE 152 Section 5 HW2: Stereo Geometry April 29, 2019 Owen Jow Stereo: two views. Why is one

CSE 152 Section 5 HW2: Stereo Geometry April 29, 2019 Owen Jow Stereo: two views. Why is one view not sufficient? 1. Depth and Disparity 3D from a single view? Ambiguity: depth lost during projection. If you multiply by the inverse

474 views • 34 slides
Stereo Vision Egon Elbre Hans Mesalu general stuff about this 3D thing why? applications

Stereo Vision Egon Elbre Hans Mesalu general stuff about this 3D thing why? applications

Stereo Vision Egon Elbre Hans Mesalu general stuff about this 3D thing why? applications games movies simulators robotics product presentations architecture visualization virtual television studios virtual presence for video

614 views • 32 slides
Faades of Interest in Street View Panoramic Sequences Andr A. Arajo, Jonas C. Sampaio,

Faades of Interest in Street View Panoramic Sequences Andr A. Arajo, Jonas C. Sampaio,

Accurate Location of Faades of Interest in Street View Panoramic Sequences Andr A. Arajo, Jonas C. Sampaio, Raphael S. Evangelista, Altobelli B. Mantuan, Leandro A. F. Fernandes {andrealvarado,revangelista}@id.uff.br,

830 views • 24 slides
Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. Andreas Butz, Dr. Julie Wagner

Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. Andreas Butz, Dr. Julie Wagner

Mensch-Maschine-Interaktion 2 Interactive Environments Prof. Dr. Andreas Butz, Dr. Julie Wagner 1 LMU Mnchen Medieninformatik Andreas Butz Mensch-Maschine-Interaktion II WS2013/14

793 views • 54 slides
Architecture exploration through FPGA acceleration Rapid System Level Design and Evaluation of

Architecture exploration through FPGA acceleration Rapid System Level Design and Evaluation of

Architecture exploration through FPGA acceleration Rapid System Level Design and Evaluation of Near Memory Fixed Function Units 11/13/2020 Maya Gokhale DMTS LLNL-PRES-816381 This work was performed under the auspices of the U.S. Department of

979 views • 59 slides
ImageNet Classification with Deep Convolutional Neural Networks Krizhevsky et. all Outline

ImageNet Classification with Deep Convolutional Neural Networks Krizhevsky et. all Outline

ImageNet Classification with Deep Convolutional Neural Networks Krizhevsky et. all Outline Introduction DataSet Architecture of the Network Reducing overfitting Learning Results Discussion Introduction

1.3k views • 31 slides

More recommend