Sensitivity Analysis of Feature Selection for Object Based Urban - - PowerPoint PPT Presentation

▶
sensitivity analysis of feature selection for object
SMART_READER_LITE
LIVE PREVIEW

Sensitivity Analysis of Feature Selection for Object Based Urban - - PowerPoint PPT Presentation

Sep 12, 2022 246 likes •533 views

Sensitivity Analysis of Feature Selection for Object Based Urban Classification Stefanos Georganos a , Michal Shimoni b , Tais Grippa a , Sabine Vanhuysse a , Moritz Lennert a , , Elonore Wolff a a Department of Geosciences, Environment and

slide-1
SLIDE 1

Sensitivity Analysis of Feature Selection for Object Based Urban Classification

Stefanos Georganosa, Michal Shimonib, Tais Grippaa, Sabine Vanhuyssea, Moritz Lennerta, , Eléonore Wolffa

a Department of Geosciences, Environment and Society (DGES), Institute for Environmental Management

and Land-Use Planning (IGEAT), Universitée libre de Bruxelles, 1050 Bruxelles, Belgium

b Signal and Image Center, Royal Military Academy, 1000 Bruxelles, Belgium

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-2
SLIDE 2

Introduction

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

2

  • Significant increase in the acquisition of very-high-resolution (VHR) satellite

data from Earth Observation (EO) missions.

  • Land Use – Land Cover (LULC) maps can be produced at unprecedented

resolutions in urban areas.

  • Significant change in the image processing approach

Pixel based classification Object based classification

slide-3
SLIDE 3

Introduction

3

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

Object-based Image Analysis (OBIA) principles:

  • 3 steps: feature extraction, segmentation, classification.
  • Several features can be computed : geometrical, textural, spectral, contextual

nature.

slide-4
SLIDE 4

Pleiades image - Ouagadougou

4

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

Capital of Burkina Faso. Area: 625 km2 Image size: 12 X 106 pixels Spatial resolution: 0.5 m Spectral resolution: 4 bands, RGB+NIR Tri-stereo option: DEM

slide-5
SLIDE 5

Feature extraction

5

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

Optical: VNIR(4bands)

  • Morphological statistics:
  • area, perimeter, compact (circle+square), fractal dimension
  • Spectral statistics:
  • Optical indices: NDVI, NDWI
  • Measures: min, max, range, mean, stddev, variance, sum, 1thquantile,

median, 3rdquantile, coefficient of variation

  • DEM:
  • Height nDSM

Total: 169 features !!

slide-6
SLIDE 6

i.segment cost

6

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-7
SLIDE 7

Contextual framework

7

  • The total amount of features that can be computed can often exceed several hundreds.
  • Highly dimensional datasets can have important effects in a classifier’s performance:
  • Reduced accuracy due to the Hughes effect
  • Increased training time
  • Overfitting from noisy/irrelevant features
  • Complex models that are difficult to interpret and transfer
  • In a large scale urban analysis other effects that have not been systematically examined in

the literature are:

  • Storage space
  • Processing time to compute all these features in an OBIA form
  • Feature Selection (FS) algorithms serves to reduce the number of predictors for each

classifiers.

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-8
SLIDE 8

Aim and Objectives

8

  • To investigate the effect of Feature Selection techniques of various types

and complexities in several state of the art classifiers in terms of:

  • Classification accuracy
  • Number of features
  • To propose a new metric for model selection that:
  • Quantifies model parsimony
  • Storage requirements
  • Processing time
  • Prediction accuracy

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-9
SLIDE 9

Methodology – Object Based Image Analysis (OBIA)

9

  • Semi-automated processing chain for OBIA – LULC mapping

 Jupyter notebook  GRASS GIS functions  Python  R IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-10
SLIDE 10

Methodology – Classification Algorithms

10

  • Support Vector Machines (SVM)
  • Random Forest (RF)
  • K-Nearest Neighbor (KNN)
  • Recursive Partitioning (RPART)
  • Extreme Gradient Boosting (Xgboost)

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-11
SLIDE 11

Classifiers

11

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

Linear Support Vector Machines (SVM)

Adapted from Burges (1998)

Random Forest (RF)

Adapted from Belgiu et al., (2016)

slide-12
SLIDE 12

Extreme Gradient Boosting (Xgboost)

12

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

The smaller the score the better the structure is

For a given data set with n examples/members and m features 𝐸 =

𝑦𝑗 − 𝑧𝑗 𝐸 = 𝑜, 𝑦𝑗 ∈ ℝ𝑛

Our tree ensemble model uses additive K function to predict the output: 𝑧

𝑗 = 𝜚 x𝑗 = 𝑔

𝑙 𝐿 𝑙=1

x𝑗 , 𝑔

𝑙∈ ℱ,

ℱ = 𝑔 𝒴 = 𝓍𝑟(𝒴) 𝑟 ∶ ℝ𝑛 → 𝑈, 𝓍 ∈ ℝ𝑛

where

slide-13
SLIDE 13

Methodology – Feature Selection Algorithms

13

  • Filters:

Correlation-Based Feature Selection (CFS)

  • Subset ranking that maximizes correlation with the dependent variable while minimizes

correlation with independent variables

  • Embedded:

Mean Decrease in Accuracy (MDA)

  • Built-in feature evaluation in decision trees that performs FS while training
  • Wrappers:

Recursive Feature Elimination (RFE)

Variable Selection Using Random Forest (VSURF)

  • Computationally intensive methods that create several models to evaluate features

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

𝐷𝐺𝑇 = 𝑁𝑓𝑠𝑗𝑢𝑡 𝑠𝑙𝑔 𝑙 + 𝑙(𝑙 − 1)𝑠

𝑔𝑔

𝑁𝐸𝐵 = 1 𝑜 𝑗 ∈ OOB 𝐽 𝑧𝑗 = 𝑔 x𝑗 − 𝑗 ∈ OOB 𝐽 𝑧𝑗 = 𝑔 x𝑗

𝑘

OOB

𝑜 𝑢=1

slide-14
SLIDE 14

Methodology – Training and Test sets

14

LULC Training Set Size Test Set Size Buildings 157 157 Swimming Pools 68 69 Asphalt road 56 56 Brown/Red Bare Soil 90 91 White/Grey Bare Soil 72 72 Trees 77 77 Mixed Bare Soil/Vegetation 76 75 Dry Vegetation 70 70 Other Vegetation 139 141 Inland Waters 75 75 Shadow 58 59

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-15
SLIDE 15

Methodology - Classification Optimization Score (COS)

15

  • Classification Optimization Score (COS):

𝐷𝑃𝑇 = (1 + 𝑏2)

𝑂𝑜∗𝑃𝐵𝑜 𝑏2∗𝑂𝑜+𝑃𝐵𝑜

  • where 𝑂𝑜 is the normalized value of the number of features of a classification model
  • OAn is the normalized overall classification accuracy across all classifiers and feature selection

methods

  • a is the weight factor

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-16
SLIDE 16

Results – Support Vector Machine

16

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-17
SLIDE 17

Results – Extreme Gradient Boosting

17

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-18
SLIDE 18

Results – Random Forest

18

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-19
SLIDE 19

Results – Recursive Partitioning

19

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-20
SLIDE 20

Results – k Nearest Neighbors

20

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-21
SLIDE 21

Results – Overall Accuracy

21

Algorithm

CFS MDA RFE VSURF All Features Xgboost 79.2 (42) 79.1 (136) 79.8* (23) 79.5* (31) 77.8 (169) RF 78.5 (117) 78.6 (84) 78.9 (22) 78.1 (38) 77.7 (169) SVM 80.1* (37) 78.8 (111) 79.2 (32) 79.7* (75) 78.1 (169) KNN 78.2* (51) 76.2* (75) 78.0* (42) 77.9* (20) 74 (169) Rpart 69.5 (53) 69.4 (123) 69.6 (34) 70.1 (49) 69.4 (169)

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-22
SLIDE 22

Results – Classification Optimization Score

22

Classification Model COS Overall Accuracy Number of Features

RFE_XGBOOST 0.982 0.798 23 CFS_SVM 0.980 0.801 38 CFS_SVM 0.975 0.800 37 CFS_SVM 0.975 0.801 39 RFE_XGBOOST 0.972 0.792 22 VSURF_XGBOOST 0.971 0.795 31 RFE_XGBOOST 0.970 0.792 25 CFS_SVM 0.970 0.797 36 CFS_SVM 0.970 0.799 40 VSURF_SVM 0.969 0.789 20

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-23
SLIDE 23

Results – Classification Optimization Score

23

  • The amount of input features in a classifier can serve as a robust surrogate

for computational time, model complexity, data storage and processing.

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-24
SLIDE 24

Results - Classification

24

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

Classification results from the highest ranked model of the COS metric (Xgboost, 23 features) for different regions in Ouagadougou. a) Industrial, b) b) planned residential and c) c) unplanned residential neighborhoods.

slide-25
SLIDE 25

Conclusions

25

  • The selection of an appropriate FS method can have a crucial

impact on the performance of ML classifier.

  • The added value of using a wide set of features might not manifest

if no or inappropriate FS is performed such as in the case of SVM, KNN and Xgboost.

  • MDA has difficulties to detect discriminant features in heavily

redundant datasets.

  • CFS was the best all around FS method.

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-26
SLIDE 26

Discussion

26

  • RFE and VSURF performed better for Classification and Regression Trees

(CART) based classifiers.

  • In terms of pure classification accuracy, SVM and Xgboost were the best

performing classifiers.

  • By using the proposed COS index, parsimonious model selection was

performed.

  • The COS metric a more intuitive evaluation measure than other, solely

accuracy based metrics such as Overall Accuracy or Kappa index.

  • Particular benefit in large scale urban classifications where the

computational burden and complexity of the models are critical factors.

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017

slide-27
SLIDE 27

27

  • Thank you!

REACT Project: http://www.react.ulb.be MAUPP Project: http://www.MAUPP.ulb.be

IV JIAAIS Workshop, UFAL-Maceió, Brazil; 10-15 November 2017