Overview of Robot Perception Prof. Yuke Zhu Fall 2020 CS391R: Robot Learning (Fall 2020) 1
Logistics Office Hours Instructor: 4-5pm Wednesdays (Zoom) or by appointment TA: 10:15-11:15am Mondays (Zoom) or by appointment Presentation Sign-Up: Deadline Today (EOD) First review due: Wednesday 9:59pm (one review: Mask-RCNN or YOLO) Student Self-Introduction CS391R: Robot Learning (Fall 2020) 2
Today’s Agenda ● What is Robot Perception? ● Robot Vision vs. Computer Vision ● Landscape of Robot Perception ○ neural network architectures ○ representation learning algorithms ○ state estimation tasks ○ embodiment and active perception ● Quick Review of Deep Learning (if time permits) CS391R: Robot Learning (Fall 2020) 3
A key challenge in Ro bot Learning is to close the perception -action loop. Robo Perceive Perceive Act Act Act Perceive [Sa et al. IROS 2014] [Levine et al. JMLR 2016] [Bohg et al. ICRA 2018] CS391R: Robot Learning (Fall 2020) 4
What is Robot Perception? Making sense of the unstructured real world… • Incomplete knowledge of objects and scene • Imperfect actions may lead to failure • Environment dynamics and other agents CS391R: Robot Learning (Fall 2020) 5
Robotic Sensors Making contact of the physical world through multimodal senses CS391R: Robot Learning (Fall 2020) 6
Robotic Sensors Making contact of the physical world through multimodal senses [Source: HKU Advanced Robotics Laboratory] CS391R: Robot Learning (Fall 2020) 7
Robot Vision vs. Computer Vision Robot vision is embodied , active , and environmentally situated . [Detectron - Facebook AI Research] [Zeng et al., IROS 2018] CS391R: Robot Learning (Fall 2020) 8
Robot Vision vs. Computer Vision Robot vision is embodied , active , and environmentally situated . ● Embodied : Robots have physical bodies and experience the world directly. Their actions are part of a dynamic with the world and have immediate feedback on their own sensation. ● Active : Robots are active perceivers. It knows why it wishes to sense, and chooses what to perceive, and determines how, when and where to achieve that perception. ● Situated : Robots are situated in the world. They do not deal with abstract descriptions, but with the here and now of the world directly influencing the behavior of the system. [Brooks 1991; Bajcsy 2018] CS391R: Robot Learning (Fall 2020) 9
Robot Perception: Landsc scape What you will learn in the chapter of Robotics and Perception 1. Modalities : neural network architectures designed for different sensory modalities 2. Representations : representation learning algorithms without strong supervision 3. Tasks : state estimation tasks for robot navigation and manipulation 4. Embodiment : active perception for embodied visual intelligence CS391R: Robot Learning (Fall 2020) 10
Robot Perception: Mo Modalities (x 1 , y 1 , z 1 ) (x 2 , y 2 , z 2 ) [Source: PointNet++; Qi et al. 2016] Pixels (from RGB cameras) Point cloud (from structure sensors) [Source: Calandra et al. 2018] [Source: Lee*, Zhu*, et al. 2018] Time series (from F/T sensors) Tactile data (from the GelSights sensors) CS391R: Robot Learning (Fall 2020) 11
Robot Perception: Mo Modalities How can we design the neural network architectures that can effectively process raw sensory data in vastly different forms? More sensory modalities in later weeks… Week 2: Object Detection (Pixels) Week 3: 3D Point Cloud CS391R: Robot Learning (Fall 2020) 12
Robot Perception: Represe sentations A fundamental problem in robot perception is to learn the proper representations of the unstructured world. [Source: Stanford CS331b] CS391R: Robot Learning (Fall 2020) 13
Robot Perception: Represe sentations “Solving a problem simply means representing it so as to make the solution transparent.” Herbert A. Simon, Sciences of the Artificial Our secret weapon? Learning CS391R: Robot Learning (Fall 2020) 14
[6.S094, MIT] CS391R: Robot Learning (Fall 2020) 15
Robot Perception: Represe sentations How can we learn representations of the world with limited supervision? Week k 3 (Thu) “Nature” “N Structural priors (inductive biases) + “N “Nurture” Interaction and movement (embodiment) Week k 4 (Tue) babies learning by playing CS391R: Robot Learning (Fall 2020) 16
Robot Perception: Represe sentations How can we learn representations that fuse multiple sensory modalities together? [The McGurk Effect, BBC] Is seeing believing? https://www.youtube.com/watch?v=2k8fHR9jKVM CS391R: Robot Learning (Fall 2020) 17
Robot Perception: Represe sentations How can we learn representations that fuse multiple sensory modalities together? Week k 4 Thu : Multimodal Sensor Fusion Reaching Alignment Insertion 1 2 3 4 5 6 1 2 3 4 5 6 [Lee*, Zhu*, et al. 2018] combining vision and force for manipulation CS391R: Robot Learning (Fall 2020) 18
Robot Perception: Tasks sks Noisy Sensory Data State Representation Perception & Robot Control & Computer Vision Decision Making CS391R: Robot Learning (Fall 2020) 19
Robot Perception: Tasks sks Localization (Week 5 Tue) Noisy Sensory Data State Representation Pose Estimation (Week 5 Thu) Visual Tracking (Week 6 Tue) Perception & Robot Control & Computer Vision Decision Making CS391R: Robot Learning (Fall 2020) 20
Robot Perception: Tasks sks Noisy Sensory Data State Representation Perception & Robot Control & http://www.probabilistic-robotics.org/ Computer Vision Decision Making CS391R: Robot Learning (Fall 2020) 21
<latexit sha1_base64="/JoDb3w1Lbpxa+WZO8WaYqU2K8=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J0IN/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRI3g7GNzO/ci1EbF6wEnC/YgOlQgFo2il+6c+9sVt+rOQVaJl5MK5Gj0y1+9QczSiCtkhrT9dwE/YxqFEzyamXGp5QNqZD3rVU0YgbP5ufOiVnVhmQMNa2FJK5+nsio5ExkyiwnRHFkVn2ZuJ/XjfF8MrPhEpS5IotFoWpJBiT2d9kIDRnKCeWUKaFvZWwEdWUoU2nZEPwl9eJa2LqndZrd1dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMf+B8/gB3hI3v</latexit> <latexit sha1_base64="7dqwAjIevYMGVrenjYTqQEQdk/c=">AB73icbVDLSgNBEOz1GeMr6tHLYBDiJexKIB6DXjxGMA9IljA7mU2GzM6uM71iCPkJLx4U8ervePNvnCR70MSChqKqm+6uIJHCoOt+O2vrG5tb27md/O7e/sFh4ei4aeJUM95gsYx1O6CGS6F4AwVK3k40p1EgeSsY3cz81iPXRsTqHscJ9yM6UCIUjKV2gGXpaceXvQKRbfszkFWiZeRImSo9wpf3X7M0ogrZJIa0/HcBP0J1SiY5N8NzU8oWxEB7xjqaIRN/5kfu+UnFulT8JY21JI5urviQmNjBlHge2MKA7NsjcT/M6KYZX/kSoJEWu2GJRmEqCMZk9T/pCc4ZybAlWthbCRtSTRnaiPI2BG/5VXSvCx7lXL1rlKsXWdx5OAUzqAEHlShBrdQhwYwkPAMr/DmPDgvzrvzsWhdc7KZE/gD5/MHgmSPow=</latexit> <latexit sha1_base64="1EdS5PnvuyIMSClqyAfSQ+0cy5M=">AB8XicbVBNS8NAEJ34WetX1aOXYBHqpSRSqMeiF48V7Ae2IWy2m3bpZhN2J2Kt/RdePCji1X/jzX/jts1BWx8MPN6bYWZekAiu0XG+rZXVtfWNzdxWfntnd2+/cHDY1HGqKGvQWMSqHRDNBJesgRwFayeKkSgQrBUMr6Z+654pzWN5i6OEeRHpSx5yStBId0np0cenBx/P/ELRKTsz2MvEzUgRMtT9wle3F9M0YhKpIFp3XCdBb0wUcirYJN9NUsIHZI+6xgqScS0N5dPLFPjdKzw1iZkmjP1N8TYxJpPYoC0xkRHOhFbyr+53VSDC+8MZdJikzS+aIwFTbG9vR9u8cVoyhGhCquLnVpgOiCEUTUt6E4C6+vEya52W3Uq7eVIq1yOHBzDCZTAhSrU4Brq0AKEp7hFd4sb1Y79bHvHXFymaO4A+szx9grJC9</latexit> <latexit sha1_base64="rcCQoP/aEiLnvtOiarySz7BnQE=">AB+3icbVDLSsNAFJ3UV62vWJduBotQUsihbosunFZwT6gDWEynbRDJw9mbqQl5lfcuFDErT/izr9x2mah1QMXDufcy73eLHgCizryisrW9sbhW3Szu7e/sH5mG5o6JEUtamkYhkzyOKCR6yNnAQrBdLRgJPsK43uZn73QcmFY/Ce5jFzAnIKOQ+pwS05JrluDp14TFx4XzqpnBhZ2euWbFq1gL4L7FzUkE5Wq75ORhGNAlYCFQpfq2FYOTEgmcCpaVBoliMaETMmJ9TUMSMOWki9szfKqVIfYjqSsEvFB/TqQkUGoWeLozIDBWq95c/M/rJ+BfOSkP4wRYSJeL/ERgiPA8CDzklEQM0IlVzfiumYSEJBx1XSIdirL/8lncuaXa817uqV5nUeRxEdoxNURTZqoCa6RS3URhRN0RN6Qa9GZjwb8b7srVg5DNH6BeMj29hwJQG</latexit> <latexit sha1_base64="TheT5UxEhslRmdBZl0X0uTguJY=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J2IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRI3g7GNzO/ci1EbF6wEnC/YgOlQgFo2il+6c+9sVt+rOQVaJl5MK5Gj0y1+9QczSiCtkhrT9dwE/YxqFEzyamXGp5QNqZD3rVU0YgbP5ufOiVnVhmQMNa2FJK5+nsio5ExkyiwnRHFkVn2ZuJ/XjfF8MrPhEpS5IotFoWpJBiT2d9kIDRnKCeWUKaFvZWwEdWUoU2nZEPwl9eJa2LqndZrd1dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMf+B8/gB0eI3t</latexit> <latexit sha1_base64="/eqKUhd47miGuwmnBPvPmbKGWM=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsN+3SzSbsToQS+hO8eFDEq7/Im/GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW/4PCofHzSNnGqGW+xWMa6G1DpVC8hQIl7ya0yiQvBNMbud+54lrI2L1iNOE+xEdKREKRtFKD+kAB+WKW3UXIOvEy0kFcjQH5a/+MGZpxBUySY3peW6CfkY1Cib5rNRPDU8om9AR71mqaMSNny1OnZELqwxJGtbCslC/T2R0ciYaRTYzoji2Kx6c/E/r5dieO1nQiUpcsWi8JUEozJ/G8yFJozlFNLKNPC3krYmGrK0KZTsiF4qy+vk/ZV1atV6/e1SuMmj6MIZ3AOl+BHRpwB01oAYMRPMrvDnSeXHenY9la8HJZ07hD5zPH2/mjeo=</latexit> Robot Perception: Tasks sks State estimation methods: Bayes Filtering bel ( x t ) x t : state z t : observation u t : action : belief p ( x t | u t , x t − 1 ) : transition model (motion model) p ( z t | x t ) : measurement model (observation model) CS391R: Robot Learning (Fall 2020) 22
Recommend
More recommend
