Evaluatjon of maritjme event detectjon against missing data - - PowerPoint PPT Presentation

▶
evaluatjon of maritjme event detectjon against missing
SMART_READER_LITE
LIVE PREVIEW

Evaluatjon of maritjme event detectjon against missing data - - PowerPoint PPT Presentation

Mar 08, 2023 372 likes •527 views

Evaluatjon of maritjme event detectjon against missing data Maximilian Zocholl 1 , Clment Iphar 1 , Manolis Pitsikalis 2 , Anne-Laure Jousselme 1 , Alexander Artikis 2,3 ,Cyril Ray 4 1 NATO STO Centre for Maritime Research and Experimentation, La

slide-1
SLIDE 1

Slide 1

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Evaluatjon of maritjme event detectjon against missing data

Maximilian Zocholl1, Clément Iphar1, Manolis Pitsikalis2, Anne-Laure Jousselme1, Alexander Artikis2,3,Cyril Ray4

1 NATO STO Centre for Maritime Research and Experimentation, La Spezia, Italy 2 NCSR Demokritos, Athens, Greece 3 University of Piraeus, Greece 4 Naval Academy Research Institute, Brest, France

slide-2
SLIDE 2

Slide 2

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

  • 1. Evaluatjon of MSI detectjon with data removal
  • 2. Maritjme Events
  • 3. Dataset variatjons
  • 4. Discussion
  • 5. Conclusions
slide-3
SLIDE 3

Slide 3

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Evaluatjon of MSI detectjon with missing data

 The interpretatjon of evaluatjon results requires a systematjc data variatjon

with domain specifjc evaluatjon criteria.

slide-4
SLIDE 4

Slide 4

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

From a Maritjme Event to a decision

 Maritjme Events  AIS data

Problems with AIS data: Data removal inspired by

variatjons of message receptjon rate, infmuenced by distance, geography, weather, transceiver, etc.

 Tasks performed on AIS data

htups://pla.co.uk/Safety/Vessel-Traffjc-Services-VTS-/About-London-VTS

https://www.acadiainsurance.com/tug-boats-vs-push-boats/

slide-5
SLIDE 5

Slide 5

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

From a maritjme dataset to an experimental input

10.5281/zenodo.1167595 Heterogeneous Integrated Dataset for Maritime Intelligence, Surveillance, and Reconnaissance. RAY, Cyril; DRÉO, Richard; CAMOSSI, Elena; JOUSSELME, Anne-Laure. 2018

slide-6
SLIDE 6

Slide 6

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Library for dataset modifications

slide-7
SLIDE 7

Slide 7

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Dataset variatjons – used modifjcatjon functjons

Assign Event Remove

μ = mode (direct/ofgset) c = column A = subset of rows V = value of the assignment α = rate of removal Ap = subset of interest N = nature of the removal q = number of events E = nature of event p = set of parameters

slide-8
SLIDE 8

Slide 8

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Dataset for experiments

slide-9
SLIDE 9

Slide 9

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Maritjme Situatjonal Indicators

slide-10
SLIDE 10

Slide 10

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Formalisatjon of Maritjme Events with RTEC

initiatedAt (gap(Vessel) = Status, T ) ←happensAt (gap_start (Vessel), T ), happensAt (coord (Vessel, Lon, Lat), T ), portDistance(Lon, Lat, Status). terminatedAt (gap(Vessel) = Status, T ) ←happensAt (gap_end (Vessel), T ).

slide-11
SLIDE 11

Slide 11

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Evaluatjon of MSI accuracy under missing data (F1)

#MSI Pattern 90% 80% 70% 8 High speed near coast 0.999 0.989 0.982 19a Moving speed 0.996 0.995 0.976 19 Underway 0.997 0.989 0.975 2 Within area 0.987 0.979 0.974 24a Tugging speed 0.996 0.989 0.971 11 Low speed 0.992 0.983 0.96 9 Unusual speed 0.981 0.957 0.933 25a SAR course 0.934 0.835 0.89 7 Changing speed 0.939 0.888 0.815 16 Gap 0.946 0.885 0.809 6 stopped 0.911 0.795 0.63 21 MAA 0.8 0.876

  •  The larger the data

degradatjon, the lower Recall, Precision and F1.

 The performance of most

event detectors decreases slower than the data volume.

slide-12
SLIDE 12

Slide 12

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Margin of MSI accuracy variatjons due to data removal

High speed near coast

Small variatjons

Movement ability afgected

Large variatjons

slide-13
SLIDE 13

Slide 13

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Speed-based simple fmuent patuerns with large variatjons in response to data removal

 Stopped

 strong decrease of TPs  strong increase of FPs

 Unusual speed

 weak decrease of TPs  weak increase of FPs

 FPs are not

expected for stopped, underway or withinArea, unusual speed.

slide-14
SLIDE 14

Slide 14

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Perspectjves

 Creatjng removal method that removes only P

detectjons.

 Creatjng detector- or task specifjc evaluatjon

metrics, e.g. to not penalize twice FN-FP pairs for changingSpeed.

slide-15
SLIDE 15

Slide 15

Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019

Conclusions

 Application of existing AIS data variation methods for

controlled data degradations

 Exemplary performance comparison of 12 maritime event

detectors capturing robustness against missing data

 Reduction of interpretation space for reasons of dropping

performance

 Perspective for future selection method of evaluation

criteria based on applications and/or classifiers