An On-Board Visual-Based Attitude Estimation System for Unmanned - - PowerPoint PPT Presentation

An On-Board Visual-Based Attitude Estimation System for Unmanned Aerial Vehicle Mapping

• M. Ridhwan Tamjis1 and Samsung Lim2

1School of Civil and Environmental Engineering,

UNSW Sydney, NSW, 2052, Australia Email: m.tamjis@student.unsw.edu.au

2School of Civil and Environmental Engineering,

UNSW Sydney, NSW, 2052, Australia Email: s.lim@unsw.edu.au

• To develop a visual-based attitude estimation

system that can be used for unmanned aerial vehicle (UAV) mapping

• To use the system as a backup of the Inertial

Measurement Unit (IMU) to provide an accurate platform/camera pose during image acquisition

• To remove dependencies on physical ground

control points (GCPs) for accurate camera pose

Objectives

• Commercial software packages are now

readily available to undertake the tasks (e.g. ERDAS, Photo modeller, Australis, Pix4D) which are good for off-board processing where computing/processing constraints are minimal

• We aim for on-board attitude estimation

and camera self-calibration

Rationale

• Use optical flow to measure the egomotion of

the on-board camera, and estimate the platform’s attitude

• Integrate optical flow with a keypoints detector

for attitude estimation and camera self- calibration

• Find the best detector for the proposed visual-

based attitude estimation system

Methodology

• VGCPs are key points on the image,

extracted from salient features

• VGCPs should be present in overlapping

images

• VGCPs are crucial to estimate the pose of

the camera/platform based on the image plane

Virtual Ground Control Points (VGCPs)

Image Plane

Image Plane Ground Scene Camera (Xc,Yc,Zc) Ground Point P (X,Y,Z) Image Point p (x,y) Focal Length, f Depth, Z

Velocity on the image plane

vx vy vz ! " # # # # # $ % & & & & & = 1 Z − f x − f y 1 ! " # # # $ % & & & × Tx Ty Tz ! " # # # # # $ % & & & & & + xy f −( f + x2 f ) y −( f + y2 f ) xy f x 1 ! " # # # # # # # # # $ % & & & & & & & & & × ωx ωy ωz ! " # # # # # $ % & & & & &

Optical flow field !

A series of tests have been conducted to determine optimal key points based on the following criteria:

• Detection rate
• Time-to-complete (TTC)
• Matching rate

Detection of key points

A fixed-wing UAV is used to acquire 249

• verlapping images over Loftus Oval, New South

Wales, Australia

• Fixed-wing UAVs
• Copters

Aerial Image Acquisition

Swinglet CAM and eBEE RTK – High altitude flight (> 100 m) DJI Phantom 3 and Parrot Bebop – Low altitude flight (<100 m) and system development test bed

Aerial images over Loftus Oval

• Harris detector
• Minimum Eigenvalue (MinEig)
• Scale Invariant Feature Transform (SIFT)
• Maximally Stable Extremal Region (MSER)
• Speeded Up Robust Feature (SURF)
• Features From Accelerated Segment Test (FAST)
• Binary Robust Scale Invariant Keypoint (BRISK)

Key Points Detectors

Two matching metrics were used to evaluate each key points detector

• Sum of absolute differences (SAD)
• Sum of squared differences (SSD)

Matching metrics

∑ = − + − + − + + ∑ = = 2 2 2 )) 2 , 1 ( ) , ( ( 1 1 ) 2 , 1 ( n n j d j y d i x g j y i x f n n i d d SSD ∑ = − + − + − + + ∑ = = 2 2 | )) 2 , 1 ( ) , ( ( | 1 1 ) 2 , 1 ( n n j d j y d i x g j y i x f n n i d d SAD

1000 2000 3000 4000 5000 6000 BRISK FAST Harris MinEig MSER SIFT SURF SSD SAD

Detected key points (mean)

MinEig > SIFT > Harris > SURF > MSER > FAST > BRISK

50 100 150 200 250 300 350 BRISK FAST Harris MinEig MSER SIFT SURF SSD SAD

Matching* key points (mean)

* Includes correct and false matching key points SIFT > MinEig > SURF > MSER > Harris > FAST > BRISK

0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 14.0% BRISK FAST Harris MinEig MSER SIFT SURF SSD SAD

Matching* rate (%)

* Includes correct and false matching key points SIFT > SURF > MSER > FAST > BRISK > MinEig > Harris

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 BRISK FAST Harris MinEig MSER SIFT SURF SSD SAD

Time-to-complete (sec)

SIFT > MinEig > MSER > Harris > SURF > BRISK > FAST

0% 20% 40% 60% 80% 100% 120% BRISK FAST Harris MinEig MSER SURF SSD SAD

Correct* matching rate (%)

Note: SIFT was not evaluated due to the difficulty of visual inspection of matching points

MinEig > Harris > BRISK > FAST > SURF > MSER * Depends on metrics

• SSD and SAD do not show consistent

statistics as SURF and MSER do not perform well in terms of SSD-based correct matching rate

• SURF shows the highest correct matching

rate with respect to SAD

SSD vs SAD

Which one is the optimal key points detector?

• SIFT provides the highest number of

matching key points, however, too many key points would increase the processing time (similar to MinEig)

• SURF shows an optimal processing time

(~ 0.2 sec) with a reasonable number of matching key points

Performance Analysis

• There exist a clustering pattern among

matching key points, which can be used for automatic classification of correct and false matching key points

Performance Analysis

• Use outlier information of the matching

key points

• Apply cross-correlation (cxy) to distinguish

between correct and false matching key points in overlapping images

Automatic Visual Identification of Correct Matches

cxy (k)= 1 n (xt−x)(yt+k−y)

t=1 n−k

∑

; 1 n (yt−y)(xt−k−x);

t=1 n+k

∑

# $ % % % & % % %

Algorithm

Automatic Visual Identification of Correct Matches

Start Identify Outlier, k Set value K = α+1 Clustering Cross-correlation Eliminate false matches End

Automatic Visual Identification for SURF key points

Sample ID Matching key points

Automatic Visual Identification for MinEig key points

Matching key points Sample ID

Automatic Visual Identification

Keypoints Statistical Analysis Mean error Min error Max error Standard Deviation SURF 0.403 0.00 4.00 0.88 MinEig 1.26 0.00 8.00 1.31

Matching key points before Automatic Visual Identification

Matching Key Points After Automatic Visual Identification

• Performance of various keypoints detectors was

evaluated with respect to detection rate, time-to- complete and matching rate

• A set of 249 aerial images taken using a fixed-wing

UAV’s camera has been evaluated

• Assessments are conducted based on the chosen

criteria, which aims to fulfill the UAV’s on-board application requirements

• It is concluded that SURF is the optimal key point

detector for on-board attitude estimation

Concluding Remarks