Evaluatjon of maritjme event detectjon against missing data - PowerPoint PPT Presentation

Evaluatjon of maritjme event detectjon against missing data Maximilian Zocholl 1 , Clément Iphar 1 , Manolis Pitsikalis 2 , Anne-Laure Jousselme 1 , Alexander Artikis 2,3 ,Cyril Ray 4 1 NATO STO Centre for Maritime Research and Experimentation, La Spezia, Italy 2 NCSR Demokritos, Athens, Greece 3 University of Piraeus, Greece 4 Naval Academy Research Institute, Brest, France Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 1

1. Evaluatjon of MSI detectjon with data removal 2. Maritjme Events 3. Dataset variatjons 4. Discussion 5. Conclusions Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 2

Evaluatjon of MSI detectjon with missing data  The interpretatjon of evaluatjon results requires a systematjc data variatjon with domain specifjc evaluatjon criteria. Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 3

From a Maritjme Event to a decision  Maritjme Events  AIS data https://www.acadiainsurance.com/tug-boats-vs-push-boats/ Problems with AIS data: Data removal inspired by variatjons of message receptjon rate, infmuenced by distance, geography, weather, transceiver, etc.  Tasks performed on AIS data htups://pla.co.uk/Safety/Vessel-Traffjc-Services-VTS-/About-London-VTS Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 4

From a maritjme dataset to an experimental input 10.5281/zenodo.1167595 Heterogeneous Integrated Dataset for Maritime Intelligence, Surveillance, and Reconnaissance. RAY, Cyril; DRÉO, Richard; CAMOSSI, Elena; JOUSSELME, Anne-Laure. 2018 Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 5

Library for dataset modifications Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 6

Dataset variatjons – used modifjcatjon functjons μ = mode (direct/ofgset) c = column Assign A = subset of rows V = value of the assignment q = number of events Event E = nature of event p = set of parameters α = rate of removal Remove A p = subset of interest N = nature of the removal Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 7

Dataset for experiments Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 8

Maritjme Situatjonal Indicators Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 9

Formalisatjon of Maritjme Events with RTEC initiatedAt (gap(Vessel) = Status, T ) ←happensAt (gap_start (Vessel), T ), happensAt (coord (Vessel, Lon, Lat), T ), portDistance(Lon, Lat, Status). terminatedAt (gap(Vessel) = Status, T ) ←happensAt (gap_end (Vessel), T ). Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 10

Evaluatjon of MSI accuracy under missing data (F1) #MSI Pattern 90% 80% 70%  The larger the data 8 High speed near coast 0.999 0.989 0.982 degradatjon, the lower 19a Moving speed 0.996 0.995 0.976 Recall, Precision and F1. 19 Underway 0.997 0.989 0.975 2 Within area 0.987 0.979 0.974  The performance of most 24a Tugging speed 0.996 0.989 0.971 event detectors decreases 11 Low speed 0.992 0.983 0.96 9 Unusual speed 0.981 0.957 0.933 slower than the data 25a SAR course 0.934 0.835 0.89 volume. 7 Changing speed 0.939 0.888 0.815 16 Gap 0.946 0.885 0.809 6 stopped 0.911 0.795 0.63 21 MAA 0.8 0.876 - Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 11

Margin of MSI accuracy variatjons due to data removal High speed near coast Movement ability afgected Large variatjons Small variatjons   Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 12

Speed-based simple fmuent patuerns with large variatjons in response to data removal  FPs are not expected for stopped, underway or withinArea, unusual speed.  Stopped  Unusual speed  strong decrease of TPs  weak decrease of TPs  strong increase of FPs  weak increase of FPs Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 13

Perspectjves  Creatjng removal method that removes only P detectjons.  Creatjng detector- or task specifjc evaluatjon metrics, e.g. to not penalize twice FN-FP pairs for changingSpeed. Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 14

Conclusions  Application of existing AIS data variation methods for controlled data degradations  Exemplary performance comparison of 12 maritime event detectors capturing robustness against missing data  Reduction of interpretation space for reasons of dropping performance  Perspective for future selection method of evaluation criteria based on applications and/or classifiers Quatic 2019, Ciudad Real, Spain, 10.- 13. September 2019 Slide 15