System Performance Assessments Radar-Based and Dual Sensor Systems - - PowerPoint PPT Presentation

Procedures for FOD Detection System Performance Assessments

Radar-Based and Dual Sensor Systems

Beth Woodworth Edwin Herricks University of Illinois

Introduction

• I will be talking today about two performance

assessment efforts.

• The initial effort began in 2004 and was

completed with assessment of an installed system on a runway at PVD in 2008.

• The second began in 2007 and was

completed in 2009.

Sensors Evaluated

• Qinetiq TarsierTM

– Radar based system – Installed at PVD

• Xsight FODetectTM

– Hybrid system – Installed at BOS

Testing of QinetiQ TarsierTM at PVD

• 94Ghz radar
• Detects FOD at ranges

up to 1 km

• 2 sensor units placed on

towers scanned Runway 5/23 at PVD

• Primary performance

criterion was a standard

• 20dBm2 target at a

range of 1km

Testing of FODetectTM at BOS

• 74GHz radar and video camera

with image processing

• Sensors replaced edge lights
• Partial installation at BOS
• Sensors covered 550 ft on

Runway 15R

• 10 surface detection units (SDUs)

– 5 SDUs on each side of the runway

• Each sensor covered

approximately 75ft x 200ft

• Primary performance criterion

detection of 0.8 in (2 cm) object

General Assessment Protocol

• Testing scheduled over approximately 12-month

period to capture varied weather conditions – PVD – June 2007- March 2008 – BOS – June 2008 - March 2009

• Targets selected to challenge each of the systems

and included

• Different sizes
• Different shapes
• Different materials
• Different hazard potentials

Detecting a Target

• Radar

– Reflectivity – energy returned to sensor – “Complex Scatterers” – most FOD has a variety of edges which have different likelihoods of being detected.

• Electro-Optical

– Background – Item surface conditions

• Color, Flat/Glossy

– Illumination –

• Lighting and shadow

Overall Goals of Testing

• Calibration – Standard Target Testing

– Uses items of known detection characteristics – Provides information on system reliability and robustness

• Performance

– Group of FOD items with different hazard potentials – Repeatedly placed at known locations in defined positions

• Blind

– Random placement and actual and simulated FOD items – This test most represents typical detection needs at an airport – Challenges system to detect a diverse and unpredictable set of items.

General Methods

• Safety –

– Because assessments were conducted at

• perational airports, safety considerations were

paramount. – Personnel were trained and briefed regularly by airport operations personnel. – Although runways were closed, safety areas were identified and drills held so that all assessment personnel were aware of their environment. – Rigorous accounting was made of all experimental FOD taken on the airport.

General Methods

• FOD log – Over 1000 items placed and retrieved!

– all items were written in a log before proceeding to the runway – after return and often during the testing the log was checked again the items in the box

• FOD box – items put in boxes with dividers in order to be easily and

quickly counted

• Marking

– Used for calibration and performance testing – Small, nearly invisible UV marks were placed on the runway – UV lights used to find markings – Locations were surveyed using a differential GPS accurate to millimeters in the X/Y plane – Surveyed locations were compared with locations provided for each detection by each technology

General Methods

• Radar

– Does NOT depend on illumination – All assessment campaigns at PVD were performed during normal 12 pm to 5 am runway closures

• Electro-Optical

– Did depend on illumination. – Testing at BOS was on Runway 15R which saw limited use allowing testing at different times. – Tests were performed at BOS during the day, at night, and at dawn and dusk to capture a range of lighting conditions.

Calibration – Standard Target Testing

• GOAL = confirm detection performance using objects with

defined detection characteristics

• Targets selected based on technology

– Metal cylinders 3in X 4.5in 1.25in with a nominal reflectivity of 0dBm2 (Large) – Metal cylinders 2in X 2.5in with a nominal reflectivity of - 10dBm2 (Medium) – Metal cylinders 1.5in x 1.25in with a nominal reflectivity of

• 20dBm2 (Small)

– Metal 2in spheres – Colored PVC cylinders 1.5in X 1.25in colored grey, white and black

Metal cylinders 3in X 4.5in 1.25in 0dBm2 (Large) Metal cylinders 2in X 2.5in

• 10dBm2

(Medium) Metal cylinders 1.5in x 1.25in

• 20dBm2

(Small) Metal spheres 2in Colored PVC cylinders 1.5in X 1.25in

Calibration Targets

Calibration- Standard Target Testing for TarsierTM

• 4 types of metal targets
• 6 transects along full length
• f the runway
• Transects 2, 3, and 4 within

1km of both radars

• Transects 1, 5, and 6 used

for system assessments

Transec t Distance from (m) Distance from (ft) Distance from S Radar 2 (m) Distance from S Radar 2 (ft) 1 1675 5494 555 1821 2 1268 4160 220 723 3 864 2834 356 1168 4 502 1647 708 2322 5 204 668 1140 3738 6 443 1454 1547 5075

Calibration for FODetectTM

• Metal cylinders 1.5in x 1.25in

tested the radar portion of the sensor

• 3 colors of PVC cylinders (white,

grey, black) and small metal cylinder

• Targets placed at varying

distances from sensor in a test rectangle (approx. 150ft x 70ft)

• Targets placed to test the system

and a single sensor

• Standard target array included 1

metal, 1 white, 1 grey, and 1 black cylinder

• Groups of targets oriented

differently

Performance Testing

• Items not selected based on sensor type as

with calibration testing

• Used examples of common items found at

airports

• Selection of items based on frequency of
• ccurrence and hazard potential
• Calibration items also used during

performance testing

• Items placed in know positions at specific

angles

Table of Standard FOD Items Used in Performance Assessments

FOD Itema Expected Hazard Frequency of Occurrence

• 1. Small Piece of Concrete

High Common

• 2. Standard Lug Nut From

Service Vehicle High Common

• 3. Roller Bearing

High Common

• 4. Chunk of Rubber

Low Common

• 5. Mechanics Wrench

High Common

• 6. Fuel Cap

High Common

• 7. Cotter Key

Moderate Common

• 8. Plastic Bottle/Bottle Cap

Low Common

• 9. Strapping Material

Moderate Common

• 10. Expansion Joint

Material Low Common

• 11. Construction Material–

Galvanized Nails or Sheetrock Screws Moderate Based on Construction Activity

• 12. Runway Infrastructure

Part–Piece of Runway Light or Signage High Uncommon

• 13. Small Fasteners

Moderate Common

• 14. Metal Strip

High Uncommon

• 15. Fiberglass Door

Moderate Common

• 16. Asphalt Chunk

High Common

This table contains the items selected by CEAT

Performance Testing Methods at PVD

• 6 arrays related to standard

target transects

• In array 5x5 grid established,

approximately 25ft between items

• 23 targets total per array (no

items set on runway lights)

• Position of items varied between

campaigns (random selection of location achieved in test planning); items in the same position in all arrays in any single campaign.

• All items rotated 45 degrees

after each scan to cover 8 cardinal points of the compass

• Line established midway

between the centerline and the edge line

• 5 identical performance

items placed at equal distances on this line across the length of test rectangle

• Items rotated so detections

at 0, 45, and 90 degrees relative center line recorded

Performance Testing Methods at BOS

Blind Testing

• Used actual FOD items

collected from runways

• Range of sizes, materials,

colors

• Over 100 items in collection
• 30 items randomly selected

for each campaign

• Placement locations chosen

at random from a grid

• Items dropped or tossed so

the orientation was random

• Up to 10 items used at one

time.

Blind Testing Grid at PVD

Typical Grouping of FOD Blind Items Blind Testing Grid of Locations at BOS

Inclement Weather Testing

• Objective to evaluate sensor performance in

variable weather conditions

• Opportunistic based on long term plan, but

scheduled adjusted to coincide with storm events.

• Access to the runways in snow was impossible

so assessments performed just after snow emergency conditions

Inclement Weather Assessment - Rain

• PVD - TarsierTM radar system
• October 2007
• Calibration targets were deployed as a rain squall moved

across the runway

• BOS – FODetectTM hybrid system
• March 2008
• One light rain event at dusk

Inclement Weather Assessment - Snow

• PVD – TarsierTM radar system

– Assessments after 2 snow events

• Jan. 24, 2008 – flurries but no accumulation
• Feb. 12, 2008 – snow changing to sleet

– Time for placement and retrieval were limited due to needs to clear the runway

Inclement Weather Assessment - Snow

• BOS – FODetectTM hybrid system

– Winter Storm Jan. 7-12 –

• Weather Conditions:

– rain, freezing rain, freezing drizzle, mist, ice pellets, fog, snow – Totaling 2.1in of wet precipitation

• Testing took place on Jan 8, 9, and 13, 2009

– Testing on Jan. 29 and 30 after snow events as well

Summary

• 2 systems – radar and hybrid
• Assessments performed at PVD and BOS
• Assessments took place in variable weather conditions
• General protocol – calibration, performance, blind –

consistent for all FOD detection system types

• Actual procedures were adapted to each senor and adjusted

for runway availability

• Targets were selected based on

– the performance claims of manufacturers – related to sensor characteristics and performance parameters