Units 11 and 12 Risk Reduction Activity MetalMapper Advanced - - PowerPoint PPT Presentation

Units 11 and 12 Risk Reduction Activity MetalMapper Advanced Classification Fort Ord, California

PRESENTED BY: DAVID EISEN JOHN JACKSON ANDY GASCHO ALISON PASKI DEAN KEISWETTER OCTOBER 22, 2015

The Team

• U.S. Army Corps of Engineers
• David Eisen
• John Jackson
• KEMRON
• Project Management
• UXO Subsurface Removal Team
• Gilbane
• Andy Gascho
• NAEVA Geophysics
• Mark Howard
• Peter Jump
• Ryan Swaffer
• Kevin Hagie
• Alison Paski
• Cora Blits
• Ben Dameron
• Acorn Science and Innovation
• Dean Keiswetter
• ESTCP/CB&I/Black Tusk Geophysics

2

ESTCP Demonstration Results

3

ESTCP Demonstration

• Primary Objective:
• Demonstrate whether large munitions such as 155mm and 8‐inch projectiles at depths to 2 feet can be

confidently classified within a challenging high metallic anomaly density background.

• Secondary Objectives:
• Demonstrate whether large munitions at depths to 4 feet can be confidently classified within a

challenging high metallic anomaly density background.

• Demonstrate if smaller munitions such as 40mm projectiles can be confidently classified within the

range of high background conditions.

4

ESTCP Grid Locations

5

Initial ROC Curve (Primary)

6

Analyst's Type

Non-TOI Small TOI Medium TOI Large TOI Total Non-TOI Small TOI Medium TOI Large TOI Total

2030 83 97 54 2264 1 24 25 2031 83 97 78 2289

Small TOI (diam<50mm) Medium TOI (50mm<=diam<100mm) Large TOI (100mm<=diam) TOI1 FtOrd BlackTusk PolMatch None MetalMapper Custom s1 v1

Number of Non-TOIs Incorrectly Classified 500 1000 1500 2000 Percent of TOIs Correctly Classified (%) 10 20 30 40 50 60 70 80 90 100 Difficult TOIs: 155mm legacy TOI1 FtOrd BlackTusk PolMatch None MetalMapper Custom s1 v1

Final(ish) ROC Curve (Primary)

7

Percent of TOIs Correctly Classified (%)

Analyst's Type

Non-TOI Small TOI Medium TOI Large TOI Total Non-TOI Small TOI Medium TOI Large TOI Total

1802 259 133 55 2249 24 24 1802 259 133 79 2273

Small TOI (diam<50mm) Medium TOI (50mm<=diam<100mm) Large TOI (100mm<=diam) TOI1 FtOrd BlackTusk PolMatch None MetalMapper Custom s2 v1

Initial ROC Curve (Secondary)

8

Analyst's Type

Non-TOI Small TOI Medium TOI Large TOI Total Non-TOI Small TOI Medium TOI Large TOI Total

1777 286 147 38 2248 10 93 6 2 111 1 2 168 6 177 1 32 33 1788 381 322 78 2569

Small TOI (diam<50mm) Medium TOI (50mm<=diam<100mm) Large TOI (100mm<=diam) TOI1or2 FtOrd BlackTusk PolMatch None MetalMapper Custom s3 v1

Number of Non-TOIs Incorrectly Classified 500 1000 1500 2000 Percent of TOIs Correctly Classified (%) 10 20 30 40 50 60 70 80 90 100 Difficult TOIs: 35mm legacy 60mm legacy 20mm legacy 35mm legacy 20mm legacy 20mm legacy 40mm legacy 35mm legacy 40mm legacy 40mm legacy + 1 more ... TOI1or2 FtOrd BlackTusk PolMatch None MetalMapper Custom s3 v1

Results and Conclusions

• 2,804 unique cued locations
• 35 total TOI 1 (100%)
• 361 total TOI 2 (350/361 = 97%)
• Achieving primary objective (large TOI to 2‐feet) = EASY!
• Achieving secondary objective 1 (large TOI to 4‐feet) = DIFFICULT but DOABLE!
• Achieving secondary objective 2 (all TOI to depth of detection)= CLOSE, but NOT POSSIBLE!
• Moving forward, need to address depth –vs‐ signal strength –vs‐ anomaly density issues
• Removal action –vs‐ risk reduction

9

Units 11 and 12 Risk Reduction Objective

10

Units 11 and 12 Risk Reduction Goals

• Remove large MEC items from planned burn

areas to address the potential risk identified for areas to be burn‐ready (with additional site preparation activities)

• Evaluate the ability of the advanced

technology to classify items of interest in high density environment in real sites as initially indicated by ESTCP demonstration

• All Quality Control and Quality Assurance

seeds and 100% of the targets of interest (TOI) were correctly classified and recovered

11

Site Background Information

• Prescribed burns within the Impact Area at Fort Ord are part of the remedy and support
• Vegetation clearance to support MEC removal actions
• Periodic burning to maintain natural habitat
• Prescribed burns originally planned for Units 11 and 12 in 2011
• Canceled due to the discovery of large MEC items on the ground surface
• Subsequent activities conducted in Units 11 and 12
• Vegetation cutting
• Surface MEC removal
• Digital geophysical mapping
• Prescribed burns rescheduled for fall of 2015

12

Risk Reduction Objective

• Reduce risk to prescribed burn personnel by removing large, near‐surface MEC that might

unintentionally detonate during prescribed burn operations

• Targets of interest (TOI)
• 155mm projectiles
• 8‐inch projectiles
• Larger MEC
• Removal depth requirements
• Outer Zone (within 436 feet of fuel breaks)
• Removal of TOI to 2‐foot depth
• Detonation presents a risk to burn personnel on

perimeter fuel break roads

• Inner Zone (greater than 436 feet from fuel breaks)
• Removal of TOI to 1‐foot depth
• Detonation presents a risk to support aircraft

flying overhead

13

Assumptions

• From the existing DGM data, it is possible to select a subset of anomalies that could represent

155mm and 8‐inch projectiles down to two‐foot depths

• Advanced geophysical classification utilizing the MetalMapper can evaluate these anomalies

and identify those that match the signal characteristics of 155mm and 8‐inch projectiles

• Depths of classified items can be predicted with high confidence

14

Risk Reduction Goals

• MetalMapper cued anomaly data analysis to model and classify each investigated anomaly
• TOI – Highly‐likely to be large MEC items (155mm projectiles, 8‐inch projectiles)
• Non‐TOI – highly‐likely to be something other than TOI
• TOI – removed prior to the commencement of burn operations
• Non‐TOI – left in place
• Targets where the acquired data does not support a confident classification decision (“cannot

analyze”) will be removed.

15

Site Layout and Anomaly Selection Procedures

16

Anomaly Selection

• Approximately 550,000 anomalies in existing EM61 detection data
• Naval Research Laboratory EM61 response data used to determine minimum response of a

155mm projectile at the required removal depths

• Outer Zone
• 2 feet below ground surface
• Select anomalies with EM61 channel 3 response values of 114mV or greater
• Inner Zone
• 1 foot below ground surface
• Select anomalies with EM61 channel 3 response values of 446mV or greater

17

MetalMapper Investigation Anomalies

Zo Zone Resp sponse se Va Value Th Thre resh shold (m (mV) Anomalies lies Unit 11 ‐ Outer 446 2,695 Unit 11 ‐ Inner 114 192 Unit 12 ‐ Outer 446 1,717 Unit 12 ‐ Inner 114 21 TO TOTA TAL 4,625

18

MetalMapper Investigation Anomalies

19

Quality Control

20

Measurement Quality Objectives

21

QC Seeding

• Blind QC seed items buried

prior to MetalMapper investigation

• Large ISO
• Average of 1 QC seed item

each day – 35 total

• Blind Seed Firewall Plan to

protect the integrity of QC seed program

22

Data Acquisition Procedures

23

Instrument Setup and Configuration

• MetalMapper advanced TEM system
• Tow vehicle with platform and sled mount
• Positioning with GPS and IMU
• Long time range collection settings
• Infield inversion settings

Initial Instrument Tests and Initial Library

• Performed tests described in the Units 11 and 12 MEC

Risk Reduction GCMR‐QAPP

• Setup and configuration
• Sensor assembly
• Software settings and configuration
• Initial IVS
• Background survey was performed
• Seeded survey compared result to large ISO in standard

library with good result

• Test pit
• 25ms data needed to establish UX‐Analyze library
• Performed measurements over large ISO, 155mm and 8

inch projectiles

Field Collection Procedures

• Twice daily IVS survey
• Initial background readings to validate location. Hourly background readings during survey.
• Collect cued reading and refine location based on infield inversion result
• Infield inversion performed in separate software from data logging software
• Ensures the MetalMapper is positioned above the target
• Provides fit location for a single source inversion result

Data Processing and Classification

27

Data Processing Steps

• UX‐Analyze Advanced Software
• ESTCP supported development, has been successfully demonstrated at multiple sites
• Special build for 25ms MetalMapper data
• Initial instrument tests and site specific library
• Initial and Daily IVS
• Daily Data Verification
• Library Validation
• Initial Classification
• Final Classification

28

Initial Instrument Tests/Establish Initial Library

• Instrument tests evaluated to confirm correct setup, configuration and operation
• Test Pit
• Were provided data from ESTCP demo test pit
• Processed data and evaluated early time gates
• Determined for large shallow items the use later time gates
• Initial IVS
• IVS library established with the initial fit location and polarizabilities
• Initial library from IVS and Test Pit
• Representative samples of expected TOI at the site
• 155mm, 8 inch, Large ISO
• Contained 44 entries with measurements of test items at different depths and orientations

29

Daily IVS

• Twice daily measurements compared to library
• Check of position (x, y, z), fit coherence and library

match metric

• Plots of results for horizontal offset
• Depth consistency
• Fit coherence
• Library match metric
• Enabled evaluation of equipment change

Data Verification

• Performed daily off‐site the day following data

collection

• Follows the workflow in the GCMR‐QAPP
• Convert data using TEM2CSV
• Import data into UX‐Analyze
• Background measurement verification
• Background correct cued survey data
• Initial modeling
• Preliminary library match
• Perform data checks
• Create initial data plots for review
• Select data for recollected and prepare list for

field crew

31

Library Validation

• Evaluation performed to determine if

additional entries should be added to the library

• Cluster analysis performed
• Selected 22 calibration digs
• 6 – TOI threshold verification
• 13 – from unknown clusters
• 2 – parameter space (size‐decay) evaluation
• Updated initial library with excavation

results

• 3 – 155mm models for ranked dig list
• 18 other munitions (non‐TOI)
• Library Validation is a separate task from

Classification Validation

32

Initial Classification

• Initial classification performed while data

collection was ongoing

• Used updated library and thresholds based on

calibration dig results

• Conservative thresholds set for ranking,

values were selected to ensure a high level of safety while adding a low number of non‐TOI digs

• Ranked list is sorted into
• Category ‐1 (Calibration Digs)
• Category 0 (Cannot Analyze)
• Category 1 (High‐Confidence TOI)
• Category 2 (Cannot Decide)
• Category 3 (Non‐TOI)

33

Key Thresholds for Classification

• Verification digs were performed on each of these

key thresholds that were used

• Library match statistic
• Category 1 (High‐Confidence TOI)
• Category 2 (Cannot Decide)
• Category 2 (Cannot Decide, Low Signal)
• Category 3 (Non‐TOI) – below library match statistic

threshold or outside limits on decay, size and signal amplitude

• Category 0 (Cannot Analyze)
• Inversion failed or cannot extract reliable betas
• Poor fit coherence
• Unreasonable depth
• High Chi2
• Modeled depth limit was set at 2m below the

sensor, well below the planned removal depth

34

Final Classification

• Performed after completion of data collection

and refined as intrusive results were received

• Combines all preliminary lists
• Final library validation – cluster and select

additional calibration digs

• Review of dig results as they come in
• Modification based on 21187 QA seed to add

Chi2 threshold – RCA‐MM001

• Modification based on 13147 to include good

library matches far from the cued and flag location – outside of the standard 40cm offset

• Review of QC and QA seeds

35

Classification Results

• 4,625 anomalies investigated with MetalMapper
• 589 anomalies identified for intrusive investigation and subsurface removal
• 22 analyst calibration digs
• 567 ranked classification digs
• Category 0 (Cannot Analyze):

38

• Category 1 (High‐Confidence TOI):

306

• Category 2 (Cannot Decide):

223

• 4,036 anomalies classified as Category 3 (Non‐TOI)
• 87.3% of targets deemed safe to be left in the ground

36

Intrusive Investigation and Results

37

Intrusive Investigation Results

• Revisions to classification were made as initial intrusive results came in with all revisions

submitted to be competed during scheduled intrusive activities

• Most TOI were recovered in Category 1 (High‐Confidence TOI)
• Good correlation between modeled data sources and intrusive results

38

Classification Recovery Summary

39

Ca Category Qu Quantity Rec Recove vered Co Correctly Cl Classified as as TO TOI % Co Correctly Classi ssifi fied Ca Category 0 C 0 Category 1 C 1 Category 2 8‐inch Projectile 2 2 100 2 155mm Projectile 235 235 100 224 11 QA Seed 24 24 100 3 21 QC Seed 35 35 100 32 3

Category 1 = High‐Confidence TOI Category 2 = Cannot Decide Category 0 = Cannot Analyze

Recovered TOI

• 296 TOI were recovered from the 589 Calibration, Category 0, Category 1, and Category 2

intrusive investigations:

• 2 8‐inch projectiles
• MEC:

2

• MD:
• 235 155mm projectiles
• MEC:

17

• MD:

218

• 35 QC seed items (large ISOs)
• 24 QA seed items (large ISOs and 155mm projectiles)
• 36 of the TOI were recovered below the 1‐ and 2‐foot depth thresholds

40

Recovered TOI

41

Non‐TOI in Category 1

• Three targets in Category 1 did not result in

the recovery of MEC or MD

• Review of the data and intrusive results

provides explanations

• 20565 – target parts that are very similar to 8‐

inch, which was best library match

• 23373 – QA seed recovered with dig 23376
• 24214 – recovered single plate‐like source,
• ther metal remains in the ground at this

location

42

TOI in Category 2

• All seeds in Category 2 were large ISOs

recovered below 24”

• Flags 22623, 23544, 23710 at 25, 26 and 27”
• One MEC 155mm
• Flag 22738 recovered at 31” and near a large

culture feature

• All others are deeper 40lb 155mms
• Below 24” recovery of 155mm
• 13191, 22515, 22754, 22934, 23547
• Noisier betas
• Between 20” and 24” recovery of 155mm
• 22361, 22408, 22578, 23816

43

Non‐TOI Munitions

• Cluster analysis resulted in identification of

several groups of targets that appeared axially‐symmetric and thick‐walled, though smaller than the risk reduction TOI

• Sampling of these clusters resulted in

recovery of non‐TOI munitions.

• 75mm projectiles
• 81mm mortar projectiles
• 4.2in mortar projectiles
• Other MD and MEC recovered include
• 4.2 inch mortars
• 105mm projectiles
• 66mm rocket
• Caches of 40mms (no single 40mms recovered)

81mm mortar projectile 4.2‐inch mortar projectile

44

Recovered Non‐TOI Munitions

• In addition to recovered TOI, 437 non‐TOI munitions items were recovered from the targets

identified for intrusive investigation:

45

Item Quan anti tity ty MEC MD 4.2‐inch Mortar Projectile 24 1 23 105mm Projectile 7 4 3 81mm Mortar Projectile 148 133 15 75mm projectile 154 14 140 66mm Rocket 1 1 40 mm Projectile 36 36 MD Components 67 67

Recovered Non‐TOI Munitions

46

Field Checks

• Requested checks of three flag locations with very high

MetalMapper signal amplitude

• 12675 – no find – metal sign
• Validation dig of category 3 target
• 21816 – QA seed (large ISO) – metal sign
• Seed recovered far from flag location
• Cannot analyze due to high chi2 value
• 23902 – QA seed (large ISO) – metal sign
• Cannot analyze due to high chi2 value

47

Interesting TOI

• 13242 – validation dig
• Dig team recovered the nose of a 155mm

projectile during validation dig operations

• This item is not considered a TOI based on

the objective of identifying and removing 155mm and 8‐inch projectiles to depths of 1 foot and 2 feet

48

Body (40lbs) Nose (5lbs)

Interesting TOI

• 13310 and 23547
• Recovered a 155mm body and a 155mm

nose about 16‐18 inches apart and recorded them as separate finds (13310/13310A and 23547/23547A)

• In both cases, the only thing in the

database that provides an idea of what’s going on are the weights (40 lbs. for the projectile bodies and 5 lbs. for the noses).

• Revised documentation to include

TOI/non‐TOI identifier along with dig results

49

Quality Assurance and Verification/Validation

50

QA Seeding

• 24 blind QA seed items buried prior to MetalMapper investigation
• Large ISO and inert 155mm projectiles
• Remaining QA seed items from the CB&I ESTCP Demonstration project were left in place and added to

the newly emplaced QA seed item list

• Integrity of QA seed program

51

Verification/Validation Investigations

• 99 Category 3 targets (High‐Confidence Non‐TOI) selected for intrusive investigation, as

described in the Data Validation Plan

• 38 library match threshold verification targets
• 11 cluster verification targets
• 25 goodness of fit (chi‐square) verification targets
• 25 QA validation targets
• No TOI were recovered from the verification and validation investigations.

52

ROC Curve

53

ROC Curve

54

Lessons Learned, Issues, and Limitations

55

Lessons Learned

• The incorporation of an IMU on the MetalMapper results in increased accuracy in positioning

the sensor compared to the EM61 array that was used for the detection survey

• Onboard inversion software proved to be very useful
• Tow vehicles have different influence on the noise level of the data, for large sources the effect

is minimal

• Use of the long time range setting was beneficial for classification of large sources in high

density environments

• Prior to the start of this project the long time MetalMapper munitions signature library was

very limited; test pit measurements were therefore performed to ensure the initial library was large enough for good classification results

• Detection data was not available to analysts during data verification to ensure the locations of

QC and QA seeds remained blind to the data analysts

• Documentation of culture features should be made available to data analysts

56

Lessons Learned

• Removal of sporadic high magnitude single

point data spikes from the data with a non‐ linear filter

• Chi2 measure of goodness of fit to model to

classify anomalies as cannot analyze

• Review of intrusive results for consistency

with predicted sources resulted in an increase in the distance from the MetalMapper sensor that large sources were classified with high confidence

57

Conclusions

58

Risk Reduction Goals Successfully Met

• Remove large MEC items from planned burn

areas to address the potential risk identified for areas to be burn‐ready (with additional site preparation activities)

• Evaluate the ability of the advanced

technology to classify items of interest in high density environment in real sites as initially indicated by ESTCP demonstration

59

Assumptions Successfully Validated

• From the existing DGM data, it is possible to

select a subset of anomalies that could represent 155mm and 8‐inch projectiles down to two‐foot depths

• Advanced geophysical classification utilizing

the MetalMapper can evaluate these anomalies and identify those that match the signal characteristics of 155mm and 8‐inch projectiles

• Depths of classified items can be predicted

with high confidence

60

Reduction in Required Effort

• Successfully reduced the number of intrusive investigations that would have been required if

all EM61 anomalies were to be intrusively investigated

• 550,000 anomalies identified in EM61 DGM data
• Investigations reduced to 4,625 anomalies that met the response characteristics of TOI
• 99.2% of DGM anomalies did not require investigation
• 688 targets intrusively investigated
• 14.9% of MetalMapper investigations required intrusive investigation
• 85.1% of MetalMapper investigations were safely left in place

61

Conclusion

• Risk reduction goals successfully met
• Results verified by the QC process
• Results validated by the QA process
• Ready for prescribed burn operations in Units 11 and 12

62

Future Advanced Classification Work

63

Questions

64