AUTONOMOUS DRONE NAVIGATION WITH DEEP LEARNING Nikolai Smolyanskiy, - - PowerPoint PPT Presentation

β–Ά
autonomous drone navigation
SMART_READER_LITE
LIVE PREVIEW

AUTONOMOUS DRONE NAVIGATION WITH DEEP LEARNING Nikolai Smolyanskiy, - - PowerPoint PPT Presentation

Feb 27, 2023 44 likes β€’400 views

AUTONOMOUS DRONE NAVIGATION WITH DEEP LEARNING Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith Project Redtail May 8, 2017 100% AUTONOMOUS FLIGHT OVER 1 KM FOREST TRAIL AT 3 M/S 2 Why autonomous path navigation? Our deep learning approach

slide-1
SLIDE 1

Nikolai Smolyanskiy, Alexey Kamenev, Jeffrey Smith

AUTONOMOUS DRONE NAVIGATION WITH DEEP LEARNING

May 8, 2017 Project Redtail

slide-2
SLIDE 2

2

100% AUTONOMOUS FLIGHT OVER 1 KM FOREST TRAIL AT 3 M/S

slide-3
SLIDE 3

3

AGENDA

Why autonomous path navigation? Our deep learning approach to navigation System overview Our deep neural network for trail navigation SLAM and obstacle avoidance

slide-4
SLIDE 4

4

WHY PATH NAVIGATION?

Industrial inspection Search and rescue Video and photography Delivery Drone racing

Drone / MAV Scenarios

slide-5
SLIDE 5

5

WHY PATH NAVIGATION?

Delivery Security Robots for hotels, hospitals, warehouses Home robots Self-driven cars

Land Robotics Scenarios

slide-6
SLIDE 6

6

DEEP LEARNING APPROACH

NVIDIA’s end-to-end self-driving car Giusti et al. 2016, IDSIA / University of Zurich Several research projects used DL and ML for navigation

Can we use vision only navigation?

slide-7
SLIDE 7

7

OUR PROTOTYPE FOR TRAIL NAVIGATION WITH DNN

slide-8
SLIDE 8

8

SIMULATION

We used software in the loop simulator (Gazebo based)

slide-9
SLIDE 9

9

PROJECT PROGRESS

slide-10
SLIDE 10

10

PROJECT TIMELINE

October

Simulator Flights

Level of Autonomy

August

DNN Prototype

December

Outdoor Flights. Control Problems

February

Forest Flights. Control and DNN problems

April

100% AI Flight

January 2017 November September March

Development 88-89% AI Flight 50% AI Flights, Oscillations, Crashes

slide-11
SLIDE 11

11

100% AUTONOMOUS FLIGHT OVER 250 METER TRAIL AT 3 M/S

slide-12
SLIDE 12

12

DATA FLOW

TrailNet DNN Camera Steering Controller Pixhawk Autopilot

Probabilities of: 3 views: Left, Center, Right 3 positions: Left, Middle, Right Next waypoint: Position Orientation Image Frame: 640x360

slide-13
SLIDE 13

13

TRAINING DATASETS

IDSIA, Swiss Alps dataset: 3 classes, 7km of trails, 45K/15K train/test sets

Automatic labelling from left, center, right camera views

Our own Pacific NW dataset: 9 classes, 6km of trails, 10K/2.5K train/test sets

Giusti et al. 2016

slide-14
SLIDE 14

14

HARDWARE SETUP

Customized 3DR Iris+ with Jetson TX1/TX2

We use a simple 720p front facing webcam as input to our DNNs Pixhawk and PX4 flight stack are used as a low level autopilot PX4FLOW with downfacing camera and Lidar are used for visual-inertial stabilization

slide-15
SLIDE 15

15

SOFTWARE ARCHITECTURE

Our runtime is a set of ROS nodes

Steering Controller PX4 / Pixhawk Autopilot TrailNet DNN SLAM to compute semi-dense maps ROS Joystick Object Detection DNN Camera

slide-16
SLIDE 16

16

CONTROL

Our control is based on waypoint setting 𝑏 = 𝛾1 ( 𝑄𝑠 𝑀𝑗𝑓π‘₯π‘ π‘—π‘•β„Žπ‘’ 𝑗𝑛𝑏𝑕𝑓 βˆ’ 𝑄𝑠 𝑀𝑗𝑓π‘₯π‘šπ‘“π‘”π‘’ 𝑗𝑛𝑏𝑕𝑓 ) + 𝛾2( 𝑄𝑠 π‘‘π‘—π‘’π‘“π‘ π‘—π‘•β„Žπ‘’ 𝑗𝑛𝑏𝑕𝑓 βˆ’ 𝑄𝑠 π‘‘π‘—π‘’π‘“π‘šπ‘“π‘”π‘’ 𝑗𝑛𝑏𝑕𝑓 ) 𝑏 βˆ’ "π‘‘π‘’π‘“π‘“π‘ π‘—π‘œπ‘•" π‘π‘œπ‘•π‘šπ‘“; 𝛾1, 𝛾2 βˆ’ "π‘ π‘“π‘π‘‘π‘’π‘—π‘π‘œ" π‘π‘œπ‘•π‘šπ‘“π‘‘ 𝑏

  • ld direction

new waypoint / direction

𝑏 > 0 π‘’π‘£π‘ π‘œπ‘‘ π‘šπ‘“π‘”π‘’, 𝑏 < 0 π‘’π‘£π‘ π‘œπ‘‘ π‘ π‘—π‘•β„Žπ‘’

slide-17
SLIDE 17

17

TRAILNET DNN

  • 1. Train ResNet-18-based network (rotation only) using large Swiss Alps dataset
  • 2. Train translation only using small PNW dataset

Input: 320x180x3 conv2_x conv3_x conv4_x conv5_x translation (3) rotation (3) Output: 6

S-RESNET-18

  • K. He et al. 2015
slide-18
SLIDE 18

18

TRAILNET DNN

Classification instead of regression Ordinary cross-entropy is not enough:

Training with custom loss

  • 1. Images may look similar and contain label noise
  • 2. Network should not be over-confident

R:1.0 L:1.0 C: 1.0

slide-19
SLIDE 19

19

TRAILNET DNN

Training with custom loss

Softmax cross entropy with label smoothing (smoothing deals with noise) Model entropy (helps to avoid model over-confidence) Cross-side penalty (improves trail side predictions) 𝑒 = 𝑏𝑠𝑕𝑛𝑏𝑦 𝒒 πœ„ = α‰Šπ‘§2βˆ’π‘’, 𝑒 = 0, 2 0, 𝑒 = 1 𝒛: 𝑑𝑝𝑔𝑒𝑛𝑏𝑦 π‘π‘£π‘’π‘žπ‘£π‘’ 𝒒: π‘‘π‘›π‘π‘π‘’β„Žπ‘“π‘’ π‘šπ‘π‘π‘“π‘šπ‘‘ 𝛽, 𝛾: π‘‘π‘‘π‘π‘šπ‘π‘ π‘‘ Loss: 𝑀 = βˆ’ ෍

𝑗

π‘žπ‘— ln 𝑧𝑗 βˆ’ 𝛽(βˆ’ ෍

𝑗

𝑧𝑗 ln 𝑧𝑗 ) + π›Ύπœ„ where

  • V. Mnih et al. 2016
slide-20
SLIDE 20

20

DNN ISSUES

slide-21
SLIDE 21

21

DNN EXPERIMENTS

NETWORK AUTONOMY ACCURACY (ROTATION) LAYERS PARAMETERS (MILLIONS) TRAIN TIME (HOURS) S-ResNet-18

100% 84% 18 10 13

SqueezeNet

98% 86% 19 1.2 8

Mini AlexNet

97% 81% 7 28 4

ResNet-18 CE

88% 92% 18 10 10

Giusti et al.

80% 79% 6 0.6 2

[K. He et al. 2015]; [F. Iandola et al. 2016]; [A. Krizhevsky et al. 2012]; [A. Giusti et al. 2016];

slide-22
SLIDE 22

22

DISTURBANCE TEST

slide-23
SLIDE 23

23

MORE TRAINING DETAILS

Data augmentation is important: flips, scale, contrast, brightness, rotation etc Undersampling for small nets, oversampling for large nets Training : Caffe + DIGITS Inference: Jetson TX-1/TX-2 with TensorRT

slide-24
SLIDE 24

24

RUNNING ON JETSON

NETWORK FP PRECISION TX-1 TIME (MSEC)

ResNet-18 32 19.0 S-ResNet-18 32 21.7 16 11.0

SqueezeNet

32 8.1 16 3.1

Mini AlexNet

32 17.0 16 7.5

YOLO Tiny

32 19.1 16 12.0

YOLO

32 115.2 16 50.4

TX-2 TIME (MSEC)

11.1 14.0 7.0 6.0 2.5 9.0 4.5 11.4 5.2 63.0 27.0

slide-25
SLIDE 25

25

OBJECT DETECTION DNN

Modified version YOLO (You Only Look Once) DNN Replaced Leaky ReLU with ReLU Trained using darknet then converted to Caffe model TrailNet and YOLO are running simultaneously in real time on Jetson

  • J. Redmon et al. 2016
slide-26
SLIDE 26

26

THE NEED FOR OBSTACLE AVOIDANCE

slide-27
SLIDE 27

27

SLAM

slide-28
SLIDE 28

28

SLAM RESULTS

dso_results.mp4 goes here

slide-29
SLIDE 29

29

PROCRUSTES ALGORITHM

Aligns two correlated point clouds Gives us real-world scale SLAM data

Find the transform

SLAM space World space

π‘₯ π‘‘π‘ˆ

slide-30
SLIDE 30

30

PIXHAWK VISUAL ODOMETRY

Optical flow sensor PX4FLOW Single-point LIDAR for height Gives 10-20% error in pose estimation

Estimating error

PX4 pose from flight in 10m square

slide-31
SLIDE 31

31

ROLLING SHUTTER

slide-32
SLIDE 32

32

SLAM FOR ROLLING SHUTTER CAMERAS

Solve for camera pose for each scanline Run time is an issue 2x - 4x slower than competing algorithms Direct Semi-dense SLAM for Rolling Shutter Cameras (J.H. Kim, C. Cadena, I. Reid) In IEEE International Conference on Robotics and Automation, ICRA 2016

slide-33
SLIDE 33

33

SEMI-DENSE MAP COMPUTE TIMES ON JETSON

TX1 CPU USAGE TX1 FPS TX2 CPU TX2 FPS DSO

3 cores @ ~60% 1.9 3 cores @ ~65% 4.1

RRD-SLAM

3 cores @ ~80% 0.2 3 cores @ ~80% 0.35

slide-34
SLIDE 34

34

  • CONCLUSIONS. FUTURE WORK

We achieved 1 km forest flights with semantic DNN Accurate depth maps are needed to avoid unexpected obstacles Visual SLAM can replace optical flow in visual-inertial stabilization Safe reinforcement learning can be used for optimal control

slide-35
SLIDE 35