hands on intro to developing vision and lidar classifier
play

Hands-on Intro To Developing Vision and LiDAR Classifier Formula - PowerPoint PPT Presentation

Feb 15, 2023 •248 likes •869 views

Hands-on Intro To Developing Vision and LiDAR Classifier Formula Student Driverless Workshop powered by Introduction Sibo Zhu Zhijian Liu Haotian Tang Perception Lead at Perception Lead at Perception Lead at MIT Driverless

  1. Hands-on Intro To Developing Vision and LiDAR Classifier Formula Student Driverless Workshop powered by

  2. Introduction Sibo Zhu Zhijian Liu Haotian Tang ▪ Perception Lead at ▪ Perception Lead at ▪ Perception Lead at MIT Driverless MIT Driverless MIT Driverless ▪ Research Assistant at ▪ PhD student at MIT ▪ PhD student at MIT MIT HAN Lab HAN Lab HAN Lab https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 2

  3. Introduction https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 3

  4. Vision Perception Task Wide Angle Wide Angle VIO VIO Camera Camera Camera Camera Stereovision Pair Stereovision Pair https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 4

  5. System Requirement ▪ Latency ▪ Maximum view-to-actuation time for emergency stop from top speed during an acceleration run ▪ Mapping Accuracy ▪ Driven by downstream mapper ▪ Horizontal Field-of-View (FOV) ▪ perceive landmarks on the inside apex of a hairpin turn ▪ Look-ahead Distance ▪ depends on the full-stack-latency and vehicle deceleration rate https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 5

  6. Depth Estimation Step2 Step3 ` ` ` Mono Step 1 ` ` ` Ankit. et al. IV’19 Keypoints Detection Perspective-n-Point (PnP) Step2 Stereo Object Detection + Stereo Matching Algorithm Al https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 6

  7. Hands-on Tutorial To Train Your Own Cone Detection Network colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 7

  8. Colab Tutorial ： colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 8

  9. Software Design - Keypoints Detection ▪ Detects seven keypoints on each YOLOv3 detection ▪ A residual NN that leverages the geometric relationship between keypoints ▪ Seven detected key points will be then used in a Perspective-n-Point (PnP) to get depth https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 9

  10. Hands-on Tutorial To Train Your Own Keypoints Detection Network colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 10

  11. Colab Tutorial ： colab.research.google.com https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 11

  12. Validation - Accuracy https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 12

  13. Validation - Latency Open Sourced here: github.com/cv-core https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 13

  14. Beyond This Computer Vision Tutorial… ▪ Change the object detection backbone from YOLOv3 to YOLOv4/ EfficientNet/etc ▪ Adding temporal information for more stable and accurate detection ▪ Temporal Shift Module: hanlab.mit.edu/projects/tsm/ ▪ Inference with TensorRT in C++ ▪ Open sourced here: github.com/cv-core ▪ Prune the full YOLO architecture for cone detection task ▪ Quantization (int8) for even faster inference https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 14

  15. LiDAR Perception https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 30.08.2020 15

  16. Thank you for your attention https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020 16

  17. Hands-on Intro to Developing Vision and LiDAR Classifiers Formula Student Driverless Workshop powered by

  18. Speaker Introduction Sibo Zhu Zhijian Liu Haotian Tang ▪ RA at MIT HAN Lab ▪ PhD student at MIT ▪ PhD student at MIT ▪ Perception Lead at ▪ Perception Lead at ▪ Perception Lead at MIT Driverless MIT Driverless MIT Driverless https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  19. LiDAR Perception Song Han Feb 22, 2018 3D LiDAR Sensor 3D Point Cloud 500k+ points: (x, y, z, intensity) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  20. LiDAR Perception Task Velodyne 32C LiDAR https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  21. Autonomous Racing Vehicle: Objectives Low Latency High Accuracy (Drive Faster) (Prevent Collisions) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  22. Autonomous Racing Vehicle: Challenges Self-Driving Cars A whole trunk of computers! We need more efficient algorithms that do not consume intensive computations. https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  23. Autonomous Racing Vehicle: Challenges Fewer Laser Fewer Longer Distance Rings on Objects Laser Points Too many Laser Too many Shorter Distance Rings on Objects Laser Points https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  24. Efficient LiDAR Perception: Bottleneck Bandwidth (GB/s) Sequential Memory Access 668 8 20x slower 167 30 Random Memory Access 8 Mult and Add SRAM MemoryDRAM Memory Off-chip DRAM access is much Random memory access is more expensive than inefficient due to the potential arithmetic operation! bank conflicts! https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  25. Voxel-Based Models: Cubically-Growing Memory 8 0 128 x 128 x 128 resolution 83 GB (Titan XP x 7 ) GPU Memory (GB) 7% information loss 6 0 4 64 x 64 x 64 resolution 0 11 GB (Titan XP x 1 ) 42% information loss 2 0 3D ShapeNets [CVPR’15] 0 VoxNet [IROS’15] 20 40 60 80 100 120 3D U-Net [MICCAI’16] Voxel Resolution https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  26. Point-Based Models: Sparsity Overheads DGCNN PointCNN SpiderCNN Ours 95,1 57,4 51,8 51,5 45,3 36,3 27,0 Runtime (%) 15,6 12,2 PointNet [CVPR’17] 4,9 2,9 0,0 PointCNN [NeurIPS’18] Irregular Access Dynamic Kernel Actual Computation DGCNN [SIGGRAPH’19] https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  27. Point-Voxel Convolution (PVConv) Voxelize Convolve Devoxelize Fuse Normalize Multi-Layer Perceptron https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  28. Point-Voxel Convolution (PVConv) Multi-Layer Perceptron Point-Based Feature Transformation (Fine-Grained) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  29. Point-Voxel Convolution (PVConv) Voxel-Based Feature Aggregation (Coarse-Grained) Voxelize Convolve Devoxelize https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  30. Point-Voxel Convolution (PVConv) Voxel-Based Feature Aggregation (Coarse-Grained) Voxelize Convolve Devoxelize Fuse Normalize Multi-Layer Perceptron Point-Based Feature Transformation (Fine-Grained) https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  31. Low Resolution with Constrained Memory Original Scene Downsampled Scene https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  32. Sparse Point-Voxel Convolution (SPVConv) Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  33. Sparse Point-Voxel Convolution (SPVConv) Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  34. Designing Efficient 3D Modules (SPVConv) trunk traffic sign trunk person cyclist person https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  35. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  36. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

  37. Searching Efficient 3D Architectures (3D-NAS) Dynamic ResBlock Dynamic ResBlock Elastic Trans. Channel Elastic Res. Channel Elastic Res. Channel … Elastic Mid. Channel Elastic Mid. Channel Elastic Res. Channel Elastic Res. Channel Sparse Voxelize Devoxelize Convolution ×N Sparse Convolution Branch Fuse Multi-Layer Perceptron Point Branch https://fsg.one/academy Author: Sibo Zhu, Zhijian Liu, Haotian Tang 30.08.2020

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI

WP3 Developing research facilities LIDAR LISA LISA: 2-channels IR-VIS lidar MILI MILI: 2-channels UV eye- safe scanning lidar RALI RALI: 6-channels multiwavelength lidar FLUORES FLUORES: fluorescence lidar OLI

278 views • 10 slides
Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js

Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js

Hands on Scala.js Li Haoyi, PNWScala 14 Nov 2014 Hands on Scala.js: Agenda Intro to Scala.js Interactive Web Pages Cross-platform libraries Client-server integration Wrap Up Intro to Scala.js Intro to Scala.js

547 views • 37 slides
CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization,

CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization,

CS4501: Introduction to Computer Vision Max-Margin Classifier, Regularization, Generalization, Momentum, Regression, Multi-label Classification / Tagging Previous Class Softmax Classifier Inference vs Training Gradient Descent (GD)

564 views • 29 slides
June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of

June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of

The multiple uses of LiDAR CDA Basin WAG June 22 nd , 2017 Melissa Hendrickson 1 What is LiDAR? 2 LiDAR Returns Distance = (Speed of Light x Time of Flight)/2 3 Correcting our Existing GIS Databases 4 Mining and Geology 5 Land Form

476 views • 28 slides
Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning?

CS6501: Deep Learning for Visual Recognition Softmax Classifier + SGD Todays Class Intro to Machine Learning What is Machine Learning? Supervised Learning: Classification with K-nearest neighbors Unsupervised Learning: Clustering with

1.36k views • 91 slides
Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research

Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research

Hands Overview Outline Existing hands Robot hands of the 80s Commercial hands Research hands Prosthetics Design issues Kinematics Compliance Sensing Actuation Control Robustness Evaluation Discussion Hands of the 80s Hirose

237 views • 22 slides
LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees

LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees

LiDAR Surveying and the UAS Phenomena: Understanding How to See the Wood for the Trees SEPTEMBER 5 - 7, 2018 Terrestrial Laser Airborne LIDAR Unmanned Aerial Platforms Scanning Systems LiDAR Types Applications Platforms

362 views • 22 slides
The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May

The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May

The role of Planes and Edges in Lidar-Inertial Integration Teresa Vidal-Calleja Montreal, May 2019 cas.uts.edu.au LIDAR - INERTIAL 3D- Lidar IMU (300kHz points) Lidar Depth (10Hz) IMU Acceleration, velocity (100Hz) Sparse

438 views • 23 slides
Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR

Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR

Introduction to Static LiDAR Scanning Presented By: Anthony Falbo P.L.S. September 2020 LiDAR Scanning Overview Scanning background Applications Strengths Weaknesses Scanning Examples Fisher Associates LiDAR = light

610 views • 20 slides
Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution

Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution

Developing a Vision for Developing a Vision for Enhanced Foodborne Illness Source Attribution Information The Role of Product Tracing: t d today and tomorrow d t Caitlin Hickey Caitlin Hickey Jennifer McEntire Jennifer McEntire Institute of

174 views • 15 slides
Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands

Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands

Hands-on Tutorial Supported by Microsoft Research The CADRE project (Val Pentchev) Hands on intro to CADRE Program (Mat Hutchinson) overview Interactive demo with packages and notebooks (Filipi Silva) CADRE fellow presentation

878 views • 65 slides
What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the

What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the

What is LiDAR? 15 07/12/2018 16 16 07/12/2018 17 17 07/12/2018 First return data from the LiDAR (from the canopy) 18 07/12/2018 LiDAR data (25cm) following initial processing (vegetation removal) Historic England (2018) Using

319 views • 30 slides
Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic

Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic

Overview Launched in 2015, Vision Zero DC is an all-hands-on-deck approach to eliminate traffic fatalities and serious injuries in Washington DC by 2024 through the use of data, education, enforcement, and engineering. Vision Zero work is

454 views • 7 slides
Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System

Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System

Lecture 01: welcome and intro what LSD and Unix have in common Hands-On Unix System Administration DeCal 2012-08-27 1 / 21 The connection Two of the most The connection Introduction famous products of Diving into the shell

585 views • 21 slides
S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES

S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES

S N O GODS HANDS AND FEET ON EARTH I S S I M C M U E VISION/MISSION/VALUES Vision Everyone in missions Mission Serving hope by sharing Gods love and message unequivocally to all peoples Values Relationships

665 views • 25 slides
Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Maximum Entropy Classifier Ensembling using Ge- netic Algorithm for NER in Bengali Asif Ekbal 1

Outline Background and Motivation Classifier Ensembling Genetic Algorithms Proposed Method of Classifier Ensemble Feature Set Used Experimental Results Conclusions Future Works Maximum Entropy Classifier Ensembling using Ge- netic

749 views • 36 slides
Defining the Purpose: The Importance of the Work Dr. Luvelle Brown, Ithaca @luvelleb Dr. Oliver

Defining the Purpose: The Importance of the Work Dr. Luvelle Brown, Ithaca @luvelleb Dr. Oliver

#NYSupts4Diversity Defining the Purpose: The Importance of the Work Dr. Luvelle Brown, Ithaca @luvelleb Dr. Oliver Robinson, Shenendehowa @OliverRobinso13 Equity and Excellence are not opposing concepts Inequities Excellence without Equity

433 views • 5 slides
Wake County Created in 1771 Making History with the Garner Fire Department Presentation by Mike

Wake County Created in 1771 Making History with the Garner Fire Department Presentation by Mike

Wake County Created in 1771 Making History with the Garner Fire Department Presentation by Mike Legeros 1792 Raleigh Map 1802 Raleigh Newspaper Ad Pattern of Fire Protection 1800s Raleigh Fire Protection 1819 First fire

431 views • 11 slides
Walkthrough In collaboration with : WAM I. Grenier, A.K. Harding, JM Casandjian, M.E. DeCesar,

Walkthrough In collaboration with : WAM I. Grenier, A.K. Harding, JM Casandjian, M.E. DeCesar,

2 papers N. Renault-Tinacci Walkthrough In collaboration with : WAM I. Grenier, A.K. Harding, JM Casandjian, M.E. DeCesar, L. Guillemot, T.J. 1 April 2016 Johnson, Q. Remy, C. Venter. Why MSPs ? Growing -ray pulsar class Clues

509 views • 21 slides
The parameterised complexity of list problems on graphs of bounded treewidth Kitty Meeks Alex

The parameterised complexity of list problems on graphs of bounded treewidth Kitty Meeks Alex

The parameterised complexity of list problems on graphs of bounded treewidth Kitty Meeks Alex Scott Mathematical Institute University of Oxford APEX 2012, Paris Vertex colouring Given a graph G = ( V , E ), : V { 1 , . . . , k } is a

771 views • 58 slides
Using BRO to tattle on other tools Patrick Cain The Cooper-Cain Group. Inc.

Using BRO to tattle on other tools Patrick Cain The Cooper-Cain Group. Inc.

Using BRO to tattle on other tools Patrick Cain The Cooper-Cain Group. Inc. pcain@coopercain.com (@BC @APWG) Using BRO ZEEK to tattle on other tools Patrick Cain The Cooper-Cain Group. Inc. pcain@coopercain.com (@BC @APWG) I dont do

642 views • 20 slides
With Zeal and Patience: The Role of the Parish Worship Commission September 15, 2020 Ms. Mary

With Zeal and Patience: The Role of the Parish Worship Commission September 15, 2020 Ms. Mary

With Zeal and Patience: The Role of the Parish Worship Commission September 15, 2020 Ms. Mary Bellman Fr. Del Staigers With Zeal and Patience September 15, 2020 ~ 7:00 - 9:00 pm Introductions Opening Prayer What Brought You to

510 views • 19 slides
Approximate Counting via Correlation Decay on Planar Graphs Yitong Yin Nanjing University Chihao

Approximate Counting via Correlation Decay on Planar Graphs Yitong Yin Nanjing University Chihao

Approximate Counting via Correlation Decay on Planar Graphs Yitong Yin Nanjing University Chihao Zhang Shanghai Jiaotong University Holant Problems (Valiant 2004) instance: = ( G ( V, E ) , { f v } v V ) graph G =( V , E ) edges:

511 views • 15 slides
ARINC 653 Introduction Credits: An Avionics Standard for Safe, Partitioned Systems

ARINC 653 Introduction Credits: An Avionics Standard for Safe, Partitioned Systems

ARINC 653 Introduction Credits: An Avionics Standard for Safe, Partitioned Systems Wind River 2008 IEEE CS Seminar Masmano et al. - ARINC-653 APEX based on XtratuM Ananda et al. - ARINC 653 API and its application An

749 views • 20 slides

More recommend