Thesis Defense Slides Data August 2016 CITATIONS READS 0 47 1 - - PDF document

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/306571931

Thesis Defense Slides

Data · August 2016

CITATIONS READS

47

1 author: Some of the authors of this publication are also working on these related projects: NIST PSIAP: Representative Public Safety Video Dataset View project UAS Airspace Integration View project Andrew Weinert Singapore-MIT Alliance

46 PUBLICATIONS 146 CITATIONS

SEE PROFILE

All content following this page was uploaded by Andrew Weinert on 25 August 2016.

The user has requested enhancement of the downloaded file.

Andrew Weinert

M.S. Thesis Defense December 1, 2014

An Information Theoretic Approach for Generating an Aircraft Avoidance Markov Decision Process

This work is sponsored under Air Force Contract FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government

M.S Thesis Defense- 2 AJW 12/1/14

• The Traffic Alert and Collision Avoidance System (TCAS),

designed in the 1970s-1980s, is the mandated manned collision avoidance system to prevent midair collisions

– Leverages vertical escape maneuvers – Implemented as hard-coded pseudocode

• The aircraft avoidance problem can be modeled as a Markov

Decision Process (MDP) and solved via dynamic programming.

– Consists of a state and action spaces, dynamic and reward models – Contrasts the historical TCAS pseudocode approach

• Current state of the art aircraft avoidance MDPs either use a

horizontal or vertical action set

– Limited by the “curse of dimensionality” and memory requirements – Literature review did not uncover any significant efforts to quantify the utility and memory requirements for state variables

Overview

M.S Thesis Defense- 3 AJW 12/1/14

• Developed simulation-based framework to explore MDPs

– Generate MDPs without coding new dynamics – Assess potential of space aggregation

• Applied an information theoretic approach to the aircraft

avoidance problem

– Quantified the information a state provides about a potential NMAC – Defined the inherent risk for each element in a state space

• Prototyped a joint horizontal and vertical action space MDPs

– Demonstrated both safety and operational feasibility – Added states to baseline MDP to change alerting behavior

Overview

Objective: Instead of selecting states by hand, develop a quantitative approach for selecting states to enable joint horizontal and vertical action space

M.S Thesis Defense- 4 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

Content Overview

M.S Thesis Defense- 5 AJW 12/1/14

Series of mid-air collisions in 1950s led to establishment of FAA in 1958 Subsequent early attempts of collision protection include strategic airspace design and tactical air traffic control

1956 Grand Canyon Mid-Air Collision

M.S Thesis Defense- 6 AJW 12/1/14

Collision protection provide independent safety net to protect against failures in

• Air Traffic Control intervention
• Pilot compliance with procedures
• Visual see-and-avoid

Need for Collision Protection Systems

Pacific Southwest Airlines 727 / Cessna 172 San Diego, California, 1978 144 fatalities (including 7 on ground) Aeroméxico DC-9 / Piper Archer Cerritos, California, 1986 82 fatalities (including 15 on ground)

M.S Thesis Defense- 7 AJW 12/1/14

• Development of capability spanned decades
• TCAS now required worldwide for manned aircraft*
• Collision avoidance is one part of a layered air traffic

management (ATM) architecture

• Many manned aircraft assumptions no longer valid for general

aviation and unmanned aircraft in new and emerging airspace

Development and Evolution of Collision Avoidance

Beacon Transponder Low-density airspace only Beacon Transponder All airspace

Early Radar Systems Beacon Collision Avoidance System (BCAS) Traffic Alert and Collision Avoidance System (TCAS) 1960s 1970s 1980s – 2000s

*Aircraft with a maximum take-off mass (MTOM) of over 5,700 kg (12,600 lb) or authorized to carry more than 19 passengers

M.S Thesis Defense- 8 AJW 12/1/14

Detect Track Evaluate Prioritize Declare Determine Command Execute

Collision Volume

Avoidance Maneuver

SAA Functional Execution

Threat Aircraft

Sense and Avoid (SAA)

Aircraft Avoidance for UAS

Primary sense and avoid system functions:

• Self separation – strategic maneuvering of the aircraft to maintain a safe separation

distance (well clear), when not receiving ATC separation services

• Collision avoidance – tactical, last-minute maneuver to avoid collision

TCAS is not (nor will be) certified for UASs:

• Historical performance assumptions not applicable to UASs
• UAS and manned aircraft have different responsibilities

in the ATM layered architecture

Self Separation Volume

“Sense and Avoid (SAA) is the capability of an unmanned aircraft to remain well clear from and avoid collisions with other airborne traffic”*

*FAA Sponsored “Sense and Avoid” Workshop Final Report, 2009.

M.S Thesis Defense- 9 AJW 12/1/14

*ICAO, Global Air Traffic Management Operational Concept, Doc 9854, 2005.

The ATM system is comprised of independent conflict management layers to mitigate collision risk with manned and UAS aircraft taking different responsibilities

UAS responsibility when

• perating “due regard”

UAS responsibility when ATC services not provided UAS responsibility when ATC services provided Strategic Conflict Management

• Airspace Organization
• Flight Planning
• Flow Management

Separation Provision

• Tactical maneuvering to

preserve separation minima

Collision Avoidance

• Last minute maneuver
• Independent self-contained

capability

Collision

Sense and Avoid

when UAS Self Separating Conflict Collision Hazard Collision Hazard Resolved Conflict Resolved

Air Traffic Management System

Layered Architecture / Protection

M.S Thesis Defense- 10 AJW 12/1/14

• Current version of TCAS cannot support the safety and
• perational requirements of the new airspace
• NextGen ATM will leverage GPS satellite navigation and other

surveillance sources enabling new procedures

• UAS integration will lead to co-existence with manned aircraft

Meeting requirements of future airspace requires major overhaul

• f TCAS and collision avoidance philosophy

NextGen Collision Avoidance

Evolving beyond TCAS

M.S Thesis Defense- 11 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 12 AJW 12/1/14

ACAS X

NextGen Collision Avoidance

• Model-based Logic optimized off-line using numerical techniques
• State and dynamic uncertainty
• Logic produced from multi-objective probability and utility models
• Balances cost of alerting vs cost of near mid-air collision
• Accounts for system uncertainty
• Variability in pilot behavior makes it difficult to predict future trajectories
• Logic not tied to legacy surveillance, enables use of new sources

M.S Thesis Defense- 13 AJW 12/1/14

ACAS X Variants

Variation User Group Surveillance Technology Advisories

ACAS Xa Current TCAS users (large aircraft) Active radar supplemented with passive Same as current TCAS ACAS Xo Users of specific

• perations

(e.g., CSPO) Active radar supplemented with passive Vertical and horizontal advisories ACAS Xp General aviation Passive only Reduced advisory set ACAS Xu Unmanned aircraft Potentially radar, EO/IR, etc. Vertical and horizontal advisories (for self-separation)

M.S Thesis Defense- 14 AJW 12/1/14

Concept Development Safety

Retain relevant legacy design features Minimize collision risk and safety hazards

ACAS X performance must be safe, operationally suitable, and acceptable to manned pilots and UAS operators Operational Suitability User Acceptability

Reduce interference with normal operations Pilots understand, like, and trust advisories

• Types
• Timing
• Roles and

responsibilities

• Current separation
• Future procedures
• Communications

bandwidth/spectrum

• Avoidance of aircraft
• Coordination
• No alerts close

to ground

Design Considerations

• Airframe airworthiness
• Mission management
• Avionics and ground

system certification

M.S Thesis Defense- 15 AJW 12/1/14

Collision Avoidance System Elements

• Intruder detection
• Position tracking
• Alert criteria
• Advisory selection
• Aural annunciation
• Advisory display

Surveillance Advisory Logic Display

IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT ELSE ITF.TAUV = -ITF.A/ITF.ADOT; IF (ITF.TAUV LT TVTHR AND ((ABS(ITF.VMD) LT G.ZTHR) OR (ITF.TAUV LT ITF.TRTRU)) THEN SET ZHIT ELSE CLEAR ZHIT IF (ZHIT EQ $TRUE AND ABS(ITF.ZDINT) GT P.MAXZDINT THEN CLEAR ZHIT IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT ELSE ITF.TAUV = -ITF.A/ITF.ADOT; IF (ITF.TAUV LT TVTHR AND ((ABS(ITF.VMD) LT G.ZTHR) OR (ITF.TAUV LT ITF.TRTRU)) THEN SET ZHIT ELSE CLEAR ZHIT IF (ZHIT EQ $TRUE AND ABS(ITF.ZDINT) GT P.MAXZDINT THEN CLEAR ZHIT IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT ELSE ITF.TAUV = -ITF.A/ITF.ADOT; IF (ITF.TAUV LT TVTHR AND ((ABS(ITF.VMD) LT G.ZTHR) OR (ITF.TAUV LT ITF.TRTRU)) THEN SET ZHIT ELSE CLEAR ZHIT IF (ZHIT EQ $TRUE AND ABS(ITF.ZDINT) GT P.MAXZDINT THEN CLEAR ZHIT

Sensor Measurements Resolution Advisory

M.S Thesis Defense- 16 AJW 12/1/14

2 3 1

A B A B B A

0.9 0.1 0.6 0.4 0.3 0.7 +5

• 10

+1

Objective is to maximize reward

• State space

– Set of all possible states

• Action space

– Set of all possible actions

• Dynamic model

– State transition probabilities

• Reward model

– Reward for making transitions

Markov Decision Process (MDP)

Framework for sequential decision problems

Both TCAS and ACAS X selected system states by hand and have not quantified the utility of each state

M.S Thesis Defense- 17 AJW 12/1/14

Expected value

• DP is an iterative process for computing the expected value

when starting from each state

• Best action can be derived directly from expected value

DP is an efficient way to solve an MDP

) , ( max ) ( ) ' ( ) , | ' ( ) , ( ) , (

'

a s Q s V s V a s s P a s R a s Q

a s

  



Dynamic Programming (DP)

M.S Thesis Defense- 18 AJW 12/1/14

Separation Provision Collision Avoidance Advisory Logic

ACAS X

Independent Actions

Vertical Actions Horizontal Actions Meets SAA Independent Actions

+ =

Horizontal and vertical logics implemented independently due to “curse of dimensionality” and memory constraints. Capability gap

• f a joint horizontal and vertical action set logic exists.

M.S Thesis Defense- 19 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 20 AJW 12/1/14

• The aircraft avoidance problem has two simple questions

– What is the risk of collision with another aircraft? – If the risk is sufficient, what is the optimal action to minimize risk?

• MDP must characterize problem with regards to NMAC*

– Safety is easily defined numerically and aircraft agonistic – Operational constraints are more abstract and aircraft specific

• Develop simulation-based framework to explore MDPs

– Generate MDPs without coding new dynamics – Identify the memory efficiency for each state – Assess potential of space aggregation

Quantifying Potential MDPs

Objective: Instead of selecting states by hand, develop a quantitative approach for selecting states to enable joint horizontal and vertical action space

*Lost of separation ± 500 ft horizontal and ± 100 ft vertically

M.S Thesis Defense- 21 AJW 12/1/14

• Numerically defined by three parameters

– NMAC is used as a surrogate for collisions – NMAC is a binary state and defined the encounter geometry at the closet point of approach (CPA) – A larger time to CPA is associated with smaller risk of collision

• Safety component is aircraft independent

– NMAC definition doesn’t change if flying manned or unmanned – Since numeric, very easy to encode into MDP

Conceptualizing Collision Avoidance

Safety Component

Parameter NMAC Definition Unit

Time to CPA Seconds Horizontal miss distance at CPA ±500 Feet Vertical miss distance at CPA ±100 Feet

M.S Thesis Defense- 22 AJW 12/1/14

• UASs integration invalidates many historical assumptions

– Operational constraints now widely vary between aircraft – TCAS assumes only manned aircraft with specific dynamics

• Operational component isn’t easily quantified numerically

– A specific state can’t be designated as “bad” or “good” – Component decomposes into three questions

1. How much does each action reduce the safety risk?

– Quantifies the relationship between the action and separation

2. What is the smallest time to initiate each action?

– Should only select an action when the NMAC risk is sufficient

3. What is the expected time to complete each action?

– Short alerts appear unneeded and long alerts impair the mission

Conceptualizing Collision Avoidance

Operational Component

M.S Thesis Defense- 23 AJW 12/1/14

• Problem composed of safety and operational elements

– The safety component quantifies the collision risk – The operational component quantifies selecting an action to minimize the risk

• 30-60s time window to “escape” dangerous and avoid collision

– Assumes ATC separation has failed and will provide no assistance – Validated using ownship’s vertical rate 𝒊𝒑 and NMAC definition*

Conceptualizing Collision Avoidance

𝒖𝒘 𝒊𝒑 = 𝟐𝟏𝟏 ft 𝒊𝒑 ft s 𝒖𝒘 𝟑𝟔. 𝟏 ft s = 𝟓 s 𝒖𝒘 𝟐𝟑. 𝟔 ft s = 𝟗 s 𝒖𝒘 𝒊𝒑 = 𝟔𝟏𝟏ft 𝒊𝒑 ft s 𝒖𝒘 𝟑𝟔. 𝟏 ft s = 𝟑𝟏s 𝒖𝒘 𝟐𝟑. 𝟔 ft s = 𝟓𝟏 s

Don’t want to just narrowly prevent an NMAC

*Lost of separation ± 500 ft horizontal and ± 100 ft vertically

M.S Thesis Defense- 24 AJW 12/1/14

• Shannon’s entropy characterizes the randomness (uncertainty)
• f an event given a state

– Entropy is superadditive, enabling independent analysis of states – Identify states with large amount of information about NMAC events

• Aircraft avoidance is already quantified by an event, NMAC

– NMAC is a binary state where NMAC = true is considered very “bad” – Entropy has sporadically been applied to aviation safety systems

Quantifying Potential MDPs

Information Theory and NMAC Entropy

𝑰 𝒀 = − 𝑸 𝒚𝒋 𝐦𝐩𝐡𝟑 𝑸 𝒚𝒋

𝒋

𝑰 = 𝑸 NMAC = 𝟐 𝐦𝐩𝐡𝟑 𝑸 𝑶𝑵𝑩𝑫 = 𝟐 + 𝑸 NMAC = 𝟏 𝐦𝐩𝐡𝟑 𝑸 NMAC = 𝟏

M.S Thesis Defense- 25 AJW 12/1/14

Quantifying Potential MDPs

Memory and Matrix Representation

• A full (dense) matrix allocates

memory for each element

– TCAS and ACAS X current representations

• A sparse matrix only stores

nonzero elements and indices

– No operations on zero elements – Potential for exponential memory savings over dense matrices

• Sparsity corresponds to loosely

coupled systems

– Typical of physical dynamics

0.01 0.1 1 10 100 1000 10000 25 50 75 100

Megabytes Required Percentage of nonzero elements

MATLAB Memory

Full Sparse

M.S Thesis Defense- 26 AJW 12/1/14

• Contrasts current ACAS X approach of calculating dynamics and

determining transitions during optimization

– Thesis approach represents dynamics as a large transition matrix – New approach leverages higher fidelity simulation dynamics

• Pair memory with NMAC entropy to measure computational

efficiency of states and MDPs

– Compare different states and discretizations – “Is a fine discretization of X better than a course discretization of Y?”

Simulation Framework

The simulation framework facilitates the generation of MDPs based on Monte Carlo simulations. MDPs can be generated and evaluated using any combination of states without code change*.

*Given computational memory constraints

M.S Thesis Defense- 27 AJW 12/1/14

• Encounter model describe the nominal encounter situation

without a collision avoidance system

– Based on continuous radar data feed from the U.S. Air Force 84th Radar Evaluation Squadron (RADES) (~15 GB of data/day) – Represented as a probabilistic Bayesian network

• Encounter models not used for MDP dynamics due to limited

number of variables

– Intended use to describe aircraft’s general behavior – Limited by curse of dimensionality (7-15 states depending on model)

• Sample models to build Monte Carlo simulation and record all

aircraft states at each time step in simulation

– Recorded significantly more data than traditional MIT LL analyses – Enabled new perspective on relationship between aircraft states

Simulation Framework

Encounter Models

M.S Thesis Defense- 28 AJW 12/1/14

• Leverages the Dynamic Distributed Dimensional Data Model

(D4M) to process Monte Carlo data into MDP transition matrices

– Uniform mathematical framework based on associate arrays – Doesn’t require a priori knowledge of data

• Identify potential high quality MDPs based on NMAC entropy

and memory requirements

– Optimize MDP and evaluate feasibility (shape optimization) – Determine if NMAC entropy is a good surrogate for algorithm feasibility and performance

Simulation Framework

Associate Arrays and D4M

D4M Feature Description and Benefits

Row store Processing efficient, enables constant time look up Sparsity Memory efficient, only stores non-empty columns Unlimited columns Processing not limited by discretizations or number of states High performance Efficient processing through parallel and distributed architecture

M.S Thesis Defense- 29 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 30 AJW 12/1/14

• The MDP requires an action space, state space, and costs
• Actions

– Explore the relationship between actions and control inputs – Identify differences between manned and unmanned performances

• States

– Quantify both the horizontal and vertical axes – Uniquely represent different encounter geometries

• Costs

– Deter behavior that increases the risk of NMAC – Minimize alert duration and rate with respect to safety

MDP Formulation

M.S Thesis Defense- 31 AJW 12/1/14

• Issue a waypoint to α ft
• Implement a pitch rate of Y deg

s

• Cause a pitch acceleration of Z ft

𝒕𝟑

Action Space

Variety of Potential Actions

• Climb to an altitude of α ft
• Climb at X ft s
• Climb to α ft at X ft s

Depending upon the aircraft, there are a wide variety of actions Some controls produce more deterministic behavior than others

M.S Thesis Defense- 32 AJW 12/1/14

• Evaluate two different action spaces to represent different

aircraft capabilities

– For simplicity, only vertical rates differ between spaces – A common action space across all aircraft isn’t feasible nor practical

Action Space

Command Description

COC Clear of conflict CL1500 Climb at 1500 ft/min DE1500 Descend at 1500 ft/min L3 Left turn at 3 deg/s R3 Right turn at 3 deg/s

Command Description

COC Clear of conflict CL750 Climb at 750 ft/min DE750 Descend at 750 ft/min L3 Left turn at 3 deg/s R3 Right turn at 3 deg/s

Manned Action Space UAS Action Space

M.S Thesis Defense- 33 AJW 12/1/14

• State space must contain sufficient information to represent

how each action influences the state-transition probabilities

• Decompose into a vertical axis and horizontal plane

– Treat each space independently – Consider state as part of an angle / magnitude vector perspective

• Aviation systems are typically a mix of coordination systems

State Space

M.S Thesis Defense- 34 AJW 12/1/14

TCAS / ACAS Xa State Space

𝜠𝒊 𝒊 𝒑 𝒊 𝒋 𝝊𝒊 𝒕𝑺𝑩

𝑻 = 𝜠𝒊, 𝒊𝒑 , 𝒊𝒋 , 𝝊𝒊, 𝒕𝑺𝑩

State Name

𝜠𝒊 Relative altitude 𝒊 𝒑 Ownship vertical rate 𝒊 𝒋 Intruder vertical rate

State Name

𝝊𝒊 Time to no horizontal separation 𝒕𝑺𝑩 Ownship RA state

Vertical States Other States

Ownship Intruder

M.S Thesis Defense- 35 AJW 12/1/14

ACAS Xu State Space

𝑻 = 𝑺𝒊, 𝜾, 𝚾, 𝐖𝐩, 𝐖𝐣, 𝝊𝒘, 𝑻𝑺𝑩

State Name

𝒔𝒊 Range to intruder 𝜾 Intruder bearing 𝚾 Intruder relative heading 𝝊𝒘 Time to no vertical separation

State Name

𝑾𝒑 Ownship speed 𝑾𝒋 Intruder speed 𝒕𝑺𝑩 Ownship RA state

Position States Other States

* Not illustrated on horizontal plane

𝜾 𝑻𝒋 𝑻𝒑 𝒔𝒊 𝚾 𝒕𝑺𝑩

Ownship Intruder

M.S Thesis Defense- 36 AJW 12/1/14

• An aggregate feature is a metamodel that combines multiple

states to reduce memory requirements

– Often generated through simple operations (i.e. sum) over states – 𝝊 is an aggregate feature that combines range and range rate

• Feature-based hard aggregation constructs a nonlinear (piecewise

constant) feature-based architecture – Partition space based on “similar” features between states – Associates original state with unique aggregate state / subset – Alternatives associate aggregate states with original space or both

Space Aggregation

*Illustration reproduced from Dynamic Programming and Optimal Control: Approximate Dynamic Programming by Dimitri P. Bertsekas

Aggregate States Features States

Feature Extraction Generate MDP States

𝝊𝒊 = 𝒔𝒊 𝒔𝒊 𝝊𝒘 = 𝚬𝐢 𝚬𝐢

M.S Thesis Defense- 37 AJW 12/1/14

Space Aggregation

Potential Memory Reduction

𝒊 𝒑 = 𝑫 𝒊 𝒋 = −𝑫 𝒊𝒑 + 𝒊𝒋 = 𝟏 𝒊 𝒑 = 𝟏 𝒊 𝒋 = 𝟏 𝒊𝒑 + 𝒊𝒋 = 𝟏 𝒊 𝒋 = 𝑫 𝒊 𝒑 = −𝑫 𝒊𝒑 + 𝒊𝒋 = 𝟏 𝒊 𝒑 ∈ −𝑫, 𝟏, 𝑫 𝒊 𝒋 ∈ −𝑫, 𝟏, 𝑫 𝒊 𝒑 ∈ −𝑫, 𝟏, 𝑫 𝒊𝒑 + 𝒊𝒋 ∈ 𝟏

𝒊𝒑 , 𝒊𝒋 requires 3 X 3 = 9 elements 𝒊𝒑 , 𝒊𝒑 + 𝒊𝒋 requires 3 X 1 = 3 elements

M.S Thesis Defense- 38 AJW 12/1/14

• Costs defined as indicator functions multiplied by some

constant

– ACAS X has leveraged advance cost techniques such as surrogate models and online costs – Alert behavior is “controlled” via the costs

• Costs remained simple to focus on state and action objectives

– Reversals and strengthens not included – Additional costs will be required for operational suitability

Costs

Cost When to apply

NMAC For states associated with an NMAC Horizontal Alert Alerting for a left or right turn Vertical Alert Alerting for a climb or descend COC Neither a horizontal or vertical alert is issued

M.S Thesis Defense- 39 AJW 12/1/14

• Develop alerting region via simulation

– Define a large “good” region compared to a small “bad” region – Leverage understanding of collision avoidance dynamics

• Quantify NMAC risk through a “NMAC horizon” binary variable

– 𝝃𝒖 𝑻 𝟐 if NMAC occurs within 𝒖 seconds 𝟏 otherwise – Ability to assign an NMAC cost for spaces without 𝚬𝐢, 𝐬𝐢

Costs

NMAC Horizon

NMAC

𝒔𝒊 𝜠𝒊

NMAC

𝒔𝒊 𝜠𝒊

Very “bad” region Very “good” region

ACAS X Approach NMAC Horizon Approach

M.S Thesis Defense- 40 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 41 AJW 12/1/14

1. Monte Carlo Simulations 2. Simulation Processing 3. Calculate State-Transitions 4. Generate Cost Matrix 5. Optimize Policy 6. Evaluate Policy

Implementation

State t

M.S Thesis Defense- 42 AJW 12/1/14

MS-57023B

• Developed to help certify TCAS Version 7.1 and currently used in FAA, DoD, and DHS studies
• Implements a point-mass dynamic model

Simulated 500,000 correlated encounters and recorded 52 simulation states each second

*Collision Avoidance System Safety Assessment Tool (CASSATT)

Monte Carlo Simulations

CASSATT*

M.S Thesis Defense- 43 AJW 12/1/14

• Process raw simulation files into associative arrays for D4M

– One time processing of 13,404 files with ≈10,000 lines per file – Stored in MATLAB .mat file and require from 0.09—1.2 Megabytes

1. Generated a sorted matrix with all unique state combinations and a kd-tree for each individual state 2. Preallocate state-transition matrix as an empty sparse matrix. 3. Filter relevant states using D4M and the nearest-neighbor discretization point for each simulation state are determined 4. Index into state-transition using nearest-neighbor indices Update state-transition matrix

Simulation Processing and State-Transition Generation

Simulation produced ≈ 6,970,000,000 numeric doubles* Previous processing capability supported ≈ 13,000,000 doubles

*To author’s knowledge: largest individual aircraft simulation data set ever generated

M.S Thesis Defense- 44 AJW 12/1/14

• Inefficient to define costs and

allocate nonzero elements for states not observed in state- transition matrix

• Using nonzero linear index

record to efficiently assign costs

– Ability to implement probabilistic costs using nonzero index record – Identifying indexes requires the most memory and computation

• By using linear indexing, cost

matrix generation requires up to a few minutes

Generate Cost Matrix

Naïve NMAC Cost Structure State-Transition observations

𝑻 = 𝜠𝒊 NMAC Cost Matrix

𝜠𝒊 𝒕′ (ft) 𝜠𝒊 𝒕 (ft)

M.S Thesis Defense- 45 AJW 12/1/14

• Optimization via DP-policy iteration using Inra MDP toolbox
• Policy post-processed dilated and eroded

– Minimal processing using diamond morphological structuring element sized less than 1% of state space – Addressed noise or lack of individual elements in state-transition

• Policy implemented as a look-up table in CASSATT and

evaluated via Monte Carlo simulation

– Safety metrics driven by NMAC – Operational metrics driven be alert rate

Optimize and Evaluate Policy

Optimization Parameter Value

Max Iterations 100 Discount value 0.9 # of CPUs 1

M.S Thesis Defense- 46 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 47 AJW 12/1/14

• Transition matrix represents the dynamics from state s

s to state s’

– Research success dependent upon generation of sufficient transition matrices from simulation data – Must be smooth without discontinuities

• 𝜠𝒊 𝒕 distribution validates ability to create transition matrices

Transition Matrix Generation

COC DES1500 CL1500 L3 R3

𝜠𝒊 𝒕 Distribution*

− 5000 10−2

Log(Pr(s)) 𝜠𝒊 𝒕 (ft)

− 2500 − 7500 − 10,000 2500 5000 7500 10,000 10−4 10−6

Altitude is independent

• f turn sense

Low probability of extreme 𝜠𝒊 with opposite vertical rate sense

*Manned action space

M.S Thesis Defense- 48 AJW 12/1/14

• 𝜠𝒊 𝒕, 𝒕′ distribution* clearly illustrates sparsity principle

– Sparsity decreases as discretization becomes more course – Most states are at least 95% sparse given a reasonable discretization

Transition Matrix Generation

𝜠𝒊 𝒕, 𝒕′ Sparsity

𝜠𝒊 𝒕 (ft) 𝜠𝒊 𝒕′ (ft)

− 10,000 10,000 − 5000 5000 − 10,000 − 5000 5000 10,000

NMAC region Unrealistic to transition from 𝜠𝒊 ≥ 𝟔𝟏𝟏𝟏 to 𝜠𝒊 ≤ −𝟔𝟏𝟏𝟏 in one second Observed transition from Monte Carlo simulation

*Manned action space

M.S Thesis Defense- 49 AJW 12/1/14

Range

𝒔𝒊 𝒔𝒕 𝚬𝒊 102 103 104 105 106 107 108 109 24 21 18 15 12 9 6 3

Entropy (bits)

1010

# of discretized points Negible difference between 𝒔𝒊, 𝒔𝒕 Horizontal plane provides more information than vertical axis 𝚬𝐢

NMAC Entropy

𝜠𝒊 𝒔𝒊 𝒔𝒕 If 𝜠𝒊 ≤ 𝝑, then 𝒔𝒕 ∼ 𝒔𝒊

M.S Thesis Defense- 50 AJW 12/1/14

Range

𝚬𝐢, 𝐬𝒊

• Fine uniform discretizations

are not feasible

– Basic 𝚬𝐢, 𝐬𝒊 spaces can use up to 100GB of memory

• Increasing discretization of

𝐬𝒊 has greater effect on NMAC entropy but also on memory required

• Finer discretizations of 𝜠𝒊

have severe diminishing returns

– Aircraft rarely deviate from cruising altitude NMAC Entropy Memory Required (MB)

M.S Thesis Defense- 51 AJW 12/1/14

Range Rates

102 103 104 105 106 107 108 101 10− 1 10− 3

Memory (MB)

102 103 104 105 106 107 108 15 10 5

Entropy (bits)

NMAC Entropy Memory Required (MB)

# of discretized points

𝒔𝒊 𝒔𝒕 𝚬𝐢

A single aircraft can’t directly control a rate, it is subject to both aircraft

Like range states, negible difference between 𝒔𝒊 , 𝒔𝒕 𝚬𝐢 = −𝟒𝟏𝟏𝟏, 𝟑𝟘𝟘𝟘, … , 𝟑𝟘𝟘𝟘, 𝟒𝟏𝟏𝟏 Basic vertical rate sense isn’t sufficient (i.e. simply knowing 𝚬𝐢 is changing isn’t very useful)

𝒔𝒕 = spherical range rate 𝒔𝒊 = horizontal range rate 𝚬𝐢 = vertical range rate

M.S Thesis Defense- 52 AJW 12/1/14

Simple 𝝊

102 103 104 9 6 3

Entropy (bits)

102 104

Memory (MB)

100 10-1 10-2 10-3 103

NMAC Entropy Memory Required (MB)

# of discretized points

𝝊𝒘,𝟐𝟏𝟏 𝝊𝒊,𝟐𝟏𝟏 𝝊𝒕,𝟔𝟏 𝝊𝒊,𝟔𝟏 𝝊𝒘,𝟔𝟏 𝝊𝒕,𝟐𝟏𝟏

Similar to range and range rate, no difference between spherical and horizontal Extending edge from 50s to 100s significantly improves NMAC entropy Space is very dense, memory requirements agnostic of different 𝝊 states and discretizations

Utility of a 𝝊 state depends upon on the discretized edges

𝝊𝒊 = 𝒔𝒊 𝒔𝒊 𝝊𝒘 = 𝚬𝐢 𝚬𝐢 𝝊𝒕 = 𝒔𝒕 𝒔𝒕

M.S Thesis Defense- 53 AJW 12/1/14

Simple 𝝊

NMAC Entropy vs. Probability

10 20 30 40 60 70 80 90 100

Probability

50 10-4 10-7 10-10 𝟔𝟏 s 𝟓𝟏 s 𝟏 s 𝟐𝟏 s 𝟑𝟏 s 𝟒𝟏 s 𝟕𝟏 s 𝟘𝟏 s 𝟗𝟏 s 𝟖𝟏 s

𝝊𝒊 𝒕′

NMAC Entropy of a given 𝝊𝒊(𝒕) to 𝝊𝒊(𝒕′)

10 20 30 40 60 70 80 90 100 2 1 ·10− 2

bits

50

Probability of transitioning to lower 𝝊𝒊 increases with the value of 𝝊𝒊(𝒕) Low 𝝊𝒊(𝒕) values are likely to transition to similar low 𝝊𝒊(𝒕′) values where randomness of an NMAC is smaller

Probability of transitioning from 𝝊𝒊(𝒕) to 𝝊𝒊(𝒕′)

𝝊𝒊 𝒕′ 𝝊𝒊 𝒕 = 𝟘𝟏 s Large 𝝊𝒊(𝒕) values provide less information about transitioning to an NMAC 𝝊𝒊 𝒕 = 𝟐𝟏 s

𝝊𝒊 = 𝒔𝒊 𝒔𝒊 𝝊𝒘 = 𝚬𝐢 𝚬𝐢 𝝊𝒕 = 𝒔𝒕 𝒔𝒕

M.S Thesis Defense- 54 AJW 12/1/14

Angles

104 105 12 9 6 3

Entropy (bits)

104 105

Memory (MB)

100 10-1 10-2 101

NMAC Entropy Memory Required (MB)

# of discretized points

𝜾𝒕 𝚬𝝎 𝝎𝑺

Angular states have no edge due to 𝟑𝝆 wrap around

Angular states are very memory intensive due to wrap around

Angular states provide significant entropy but with large memory requirements 𝜾𝒕 yields the least NMAC entropy but doesn’t require the least memory

𝜠𝝎 = relative heading 𝝎𝑺 = angle of resultant 𝜾𝒕 = inclination angle

M.S Thesis Defense- 55 AJW 12/1/14

• Inclination angle (θs) is not a good

state for an aircraft avoidance MDP

– Negligible change between 𝜠𝒊 states when 𝒔𝒕 is sufficiently large 𝒔𝒕 ≥ 𝟕𝟏𝟏𝟏 ft

• Generates some of the lowest

NMAC entropy overall

• Mathematical intuition matches

NMAC entropy result

Angles

Inclination Angle 𝜾𝒕

00 1500

𝒔𝒕 (ft)

4500 6000 45 90 135 180

(deg)

− 6000 − 3000 3000 6000 3000

𝜠𝒊 𝒔𝒕, 𝜾𝒕 Distribution

𝜾𝒕 = 𝜹 𝜾𝒕 ≪ 𝜹

Small 𝒔𝒕 Large 𝒔𝒕

𝜠𝒊 = 𝑫 𝜠𝒊 = 𝑫

M.S Thesis Defense- 56 AJW 12/1/14

Airspeed

𝒘𝒑 𝒘𝒋

∗

= 𝟐 𝒘𝒑 𝒘𝒋

≥𝟐

𝒘𝒑 𝒘𝒋 + 𝟐 𝒘𝒑 𝒘𝒋

<𝟐 − 𝒘𝒋 𝒘𝒑

101 102 103 104 105 106 107

Entropy (bits)

15 12 9 6 3

𝒘𝒑 𝒘𝒋 ∗ shows that

complicated isn’t necessary better Aggregating operation has little effect 𝑺 yields greatest entropy

NMAC Entropy

101 102 103 106 107

Memory (MB)

104 105 101 10-1 10-3

Memory Required

𝑺 requires more memory

𝑺 𝒘𝒑 𝒘𝒑 + 𝒘𝒋 𝒘𝒑 𝒘𝒑 + 𝒘𝒋 𝒘𝒑 − 𝒘𝒋 𝒘𝒑 − 𝒘𝒋 𝒘𝒑 𝒘𝒋

∗

# of discretized points

Difficult to identify “similar” features of airspeed states

𝒘𝟏 = ownship′s airspeed 𝒘𝒋 = intruder′s airspeed

M.S Thesis Defense- 57 AJW 12/1/14

Vertical Rates

𝒊𝒑 𝒊𝒋 𝒊𝒑 𝒊𝒋 𝒊𝒑 + 𝒊𝒋 𝒊𝒑 + 𝒊𝒋 102 104 105

Memory (MB)

103 10-1 10-2 10-3

Memory Required (MB)

# of discretized points No difference between 𝒊𝒑 and 𝒊𝒋 indicates that an individual perspective isn’t as useful as a “system” view

102 103 104 105 3 6

Entropy (bits)

NMAC Entropy

Aggregate features require less memory given discretization Aggregate features yield greater NMAC entropy

Significant potential for adopting aggregate features

𝒊𝒑 = ownship′s vertical rate 𝒊𝒋 = intruder′s vertical rate

M.S Thesis Defense- 58 AJW 12/1/14

Vertical Rates

𝚬𝐢, 𝒊𝒑

• vs. 𝚬𝐢, 𝒊𝒑

+ 𝒊𝒋

NMAC Entropy Memory Required (MB)

Aggregate feature preserves more NMAC entropy as 𝚬𝐢 becomes more course Similar memory requirements for 𝚬𝐢, 𝒊𝒑 and 𝚬𝐢, 𝒊𝒑 + 𝒊𝒋 Memory requirements grow faster than NMAC entropy. Range grows from (8 bits, 𝝑 MB) to (12 bits, 24 MB)

𝚬𝐢 = vertical separation 𝒊𝒑 + 𝒊𝒋 = vertical rates sum

M.S Thesis Defense- 59 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 60 AJW 12/1/14

Manned 𝚬𝐢, 𝐬𝐢

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 10-2

𝝃𝟐𝟔

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

𝝃𝟒𝟏 𝝃𝟓𝟔 DES1500 CL1500 L3 R3

M.S Thesis Defense- 61 AJW 12/1/14

Manned 𝚬𝐢, 𝐬𝐢

NMAC Risk Across 𝐬𝒊

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 10-2

𝝃𝟐𝟔

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

𝝃𝟒𝟏 𝝃𝟓𝟔 CL1500 R3

Horizontal maneuvers reduce risk along 𝐬𝒊 better As horizon shrinks, so does the region of NMAC risk

M.S Thesis Defense- 62 AJW 12/1/14

Manned 𝚬𝐢, 𝐬𝐢

NMAC Risk Across 𝚬𝐢

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 10-2

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

𝝃𝟓𝟔 DES1500 CL1500 L3 R3

NMAC risk is dependent upon the vertical action. Climbing when 𝚬𝐢 ≤ 𝟏 or descending when 𝚬𝐢 ≥ 𝟏 is more risky Risk along the vertical axis is agnostic to the type of horizontal maneuver

M.S Thesis Defense- 63 AJW 12/1/14

U𝐁𝐓 𝚬𝐢, 𝐬𝐢

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 10-2

𝝃𝟐𝟔

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

𝝃𝟒𝟏 𝝃𝟓𝟔 DES750 CL750 L3 R3

M.S Thesis Defense- 64 AJW 12/1/14

• Compare risk between action spaces using convex hulls of 𝝃𝒖
• UAS’s limited performance significantly changes risk region

– Larger 𝚬𝐢 values less risky…takes longer for an NMAC to transpire – Vertical actions have less of an effect on risk along 𝚬𝐢…intruder can usually maneuver vertically greater

Comparing Manned and UAS NMAC Risk Horizons of 𝚬𝐢, 𝐬𝐢

1 2 3 4 5 ·104

𝜠𝒊 (ft)

2500 1250

• 1250
• 2500

𝒔𝒊(ft)

1 2 3 4 5

𝜠𝒊 (ft)

2500 1250

• 1250
• 2500

·104

𝒔𝒊 (ft) Manned action set (CL1500, R3) UAS action set (CL750, R3)

𝝃𝟒𝟏 : R3 / R3 𝝃𝟒𝟏 : CL1500 / CL750

M.S Thesis Defense- 65 AJW 12/1/14

• 𝚬𝐢, 𝐬𝐢 can’t describe

encounter geometries

– Corresponds to a large and unacceptable alerting region

• Adding states increases

NMAC entropy

– More information about NMAC risk – Change alerting behavior

• Complete MDP must

include angular and rate states

Adding States

(0, 1) (− 1, 0) (1, 0) (0, − 1) 𝟒𝝆 𝟓 𝝆 𝟑 𝝆 𝟓 𝟔𝝆 𝟓 𝟒𝝆 𝟑 𝟖𝝆 𝟓 𝝆 𝟑𝝆 𝟏° 𝟓𝟔° 𝟘𝟏° 𝟐𝟒𝟔° 𝟐𝟗𝟏° 𝟑𝟑𝟔° 𝟑𝟖𝟏° 𝟒𝟐𝟔° 𝒚 𝒛

M.S Thesis Defense- 66 AJW 12/1/14

• Select states to add to state space based on NMAC entropy gain

– Adding states will increase NMAC entropy – Select state with reasonable tradeoff between gain and increase in required memory – Approach can be automated but was not

• Gain for states were similar between action sets

Adding States

NMAC Entropy Gain

°)

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Gain (bits)

1 2 3 4

Manned Action Set (±1500, 3°)

Memory Increase Factor 𝚬𝝎 = 𝟏°, 𝟒°, … , 𝟒𝟕𝟏° 𝜾𝒕 = 𝟏°, 𝟒°, … , 𝟐𝟗𝟏° 𝒔𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟔𝟏𝟏 High gain but significant memory increase High gain with reasonable memory increase Marginal information gain Horizontal Vertical

𝒔𝒊 = horizontal range rate 𝜠𝝎 = relative heading 𝜾𝒕 = inclination angle

M.S Thesis Defense- 67 AJW 12/1/14

• 𝚬𝐢, 𝐬𝐢, 𝒔𝒊

can identify if encounter is becoming riskier

– Even course discretizations of 𝒔𝒊 are useful – Lacks information to fully describe geometry

Adding States

UAS Range rate 𝒔𝒊 example

𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊 𝜠𝒊

2500 1250

• 1250
• 2500

1 2 3 4 ·104

𝒔𝒊

𝒔𝒊 = 𝟑𝟔𝟏, 𝟔𝟏𝟏 CL750 R3 𝒔𝒊 = −𝟔𝟏𝟏, −𝟑𝟔𝟏

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 10-2

If aircraft are moving away from each other and an NMAC, then risk is low Risk is significantly greater if aircraft are moving towards each other

M.S Thesis Defense- 68 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 69 AJW 12/1/14

Metrics

• Risk Ratio*: the relative benefit of equipping with the system

– The primary safety metric used for aviation safety evaluations – 5NMAC risk ratio is the relative benefit protecting a volume 5X an NMAC – Risk ratio components:

• Unresolved: NMAC occurs with and without SAA
• Induced: NMAC induced by SAA that would not occur otherwise
• Alert Duration: the average alert duration

– Consistently long alerts are unacceptable for operational use – Conservative logics will alert when risk is relatively low with long durations

Risk Ratio = P(NMAC with Sense and Avoid | Encounter) P(NMAC without Sense and Avoid | Encounter) < 1 System increases safety > 1 System creates safety hazard = 1 System has no net affect on safety

• –

A logic and corresponding MDP is considered feasible if the NMAC risk ratio is less than 0.1 with an average alert duration of 60 or less seconds

M.S Thesis Defense- 70 AJW 12/1/14

• Optimal policy strongly favors vertical maneuvers

– Challenging to determine the effect of a horizontal maneuver – Reducing horizontal maneuver costs increases the number of horizontal maneuvers

Baseline 𝚬𝐢, 𝐬𝐢

Policy Generation

𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏

Very “long” alert behavior along 𝒔𝒊 axis

Policy defined by 𝝃𝟒𝟏 and equal alert costs

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range

M.S Thesis Defense- 71 AJW 12/1/14

𝝃𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s)

15 0.3851 0.2474 25 30 0.1607 0.0177 150 45 0.0665 0.0181 220

Baseline 𝚬𝐢, 𝐬𝐢

Evaluation

• A 𝚬𝐢, 𝐬𝐢 MDP produces a conservative logic
• At reasonable NMAC risk horizons, can protect against most

NMAC encounters but with unacceptable alerting behavior

• Reducing time horizon to 15s and relying on “last-second”

maneuvers minimizes alert duration but is unsafe

– Resolves many “easy” NMAC encounters – Last-second maneuvers also protect larger 5NMAC volume

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range

M.S Thesis Defense- 72 AJW 12/1/14

𝚬𝐢, 𝐬𝐢, 𝒔𝒊

Policy Generation

𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏 𝒔𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟓𝟏𝟏, 𝟔𝟏𝟏

• Inclusion of 𝒔𝒊

significantly changes alerting behavior

– Still favors vertical maneuvers (lack of angular information) – Policy requires 1.146 MB of memory

Even coarse 𝒔𝒊 discretizations substantially change the alerting behavior

Policy defined by 𝝃𝟒𝟏 and equal alert costs

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range 𝒔𝒊 = horizontal range rate

M.S Thesis Defense- 73 AJW 12/1/14

𝝃𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s)

15 0.2134 0.1843 14 30 0.1356 .1025 33 45 0.0733 .0562 50

𝚬𝐢, 𝐬𝐢, 𝒔𝒊

Evaluation

• MDP produces feasible collision avoidance logic
• Demonstrates that defining the alert volume, an operational

component, as a function of 𝝃𝒖, a safety component, is feasible

• Additional states required for operational use

– Lack of angular states limits the ability to characterize encounter geometry – Last-second maneuvers also protect larger 5NMAC volume

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range 𝒔𝒊 = horizontal range rate

M.S Thesis Defense- 74 AJW 12/1/14

• MDP with 1,035,045 unique combinations of five geometric

states and one policy state

– State-transition matrix has 1,071,318,152,025 elements – Evaluated optimal policy requires 41 MB of memory – State-transition matrix can require tens of GBs of memory

• Inclusion of angular states leads to more horizontal maneuvers

considered optimal but policies still favors vertical maneuvers

𝚬𝐢, 𝐬𝐢, 𝒔𝒊 , 𝜷, 𝒊𝒑 + 𝒊𝒋

Policy Generation

𝚬𝐢 = −𝟑𝟏𝟏𝟏, −𝟐𝟘𝟔𝟏, … , 𝟐𝟘𝟔𝟏, 𝟑𝟏𝟏𝟏 𝒔𝒊 = 𝟏, 𝟔𝟏𝟏, … , 𝟐𝟘𝟔𝟏𝟏, 𝟑𝟏𝟏𝟏𝟏 𝒔𝒊 = −𝟔𝟏𝟏, −𝟓𝟏𝟏, … , 𝟓𝟏𝟏, 𝟔𝟏𝟏 𝜷 = −𝟐𝟗𝟏, −𝟐𝟒𝟔, … , 𝟐𝟒𝟔, 𝟐𝟗𝟏 𝒊𝒑 + 𝒊𝒋 = −𝟔𝟏, 𝟑𝟔, … , 𝟑𝟔, 𝟔𝟏

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range 𝒔𝒊 = horizontal range rate 𝜷 = bearing of intruder 𝒊𝒑 + 𝒊𝒋 = vertical rates sum

M.S Thesis Defense- 75 AJW 12/1/14

𝝃𝒖 Horizon (s) 5NMAC Risk Ratio NMAC Risk Ratio Mean Alert Duration (s)

15 0.2031 .1758 20 30 0.1256 .0834 36 45 0.0937 .0651 55

𝚬𝐢, 𝐬𝐢, 𝒔𝒊 , 𝜷, 𝒊𝒑 + 𝒊𝒋

Evaluation

• Feasibility demonstrates the utility of “system” perspective

states and benefits of not decomposing into 𝝊 states

• Regardless of additional states, “last-second” logics defined by

𝝃𝟐𝟔 not safety feasible

• Continue to add states via NMAC entropy approach but difficult

to optimize with reasonable discretizations

– Tradeoff between state flexibility and efficiency

𝚬𝐢 = vertical separation 𝒔𝒊 = horizontal range 𝒔𝒊 = horizontal range rate 𝜷 = bearing of intruder 𝒊𝒑 + 𝒊𝒋 = vertical rates sum

M.S Thesis Defense- 76 AJW 12/1/14

• History of Aircraft Avoidance
• MDP for Collision Avoidance
• Problem Conceptualization
• MDP Formulation
• Implementation
• Entropy and Memory Results
• NMAC Horizon Results
• Optimization and Evaluation
• Conclusion

M.S Thesis Defense- 77 AJW 12/1/14

• Applied an information theoretic approach to the aircraft

avoidance problem

– Quantified the information a state provides about a potential NMAC – Defined the inherent risk for each element in a state space

• Demonstrated an alternative simulations-based approach to

solving collision avoidance MDPs

– Provides more flexibility because dynamics are observed, not coded – Produced similar alerting regions as ACAS X

• Prototyped a joint horizontal and vertical action space MDPs

– Illustrated preference for vertical maneuvers – Added states to baseline MDP to reach operational feasibility; safety feasibility can be achieved with a very simple state space ( 𝚬𝐢, 𝐬𝐢 )

Summary of Contributions

M.S Thesis Defense- 78 AJW 12/1/14

• Extremely large state-transition matrices are memory intensive

– Even with sparse matrices, 100’s GBs of memory could be required – Memory limits constrained the size of the MDP state space

• Monte Carlo simulations and evaluations reliant on super

computing architecture and systems

– System maintenance led to weeks of no ability to calculating NMAC entropy, NMAC horizon, or evaluate policies – Code speed optimization took longer than anticipated (ever 0.1 second counts!)

• Difficult to elegantly encourage horizontal maneuvers

– No priori analysis on relationship between different maneuvers – Unable to develop aggregate feature that quantifies angular information at reduced memory requirements

What went wrong

M.S Thesis Defense- 79 AJW 12/1/14

• Prototype simulation-based MDPs as a traditional MDP DP and

solve for dynamics during optimization

– No longer need to hold state-transition matrix in memory – Will lead to faster optimization times

• Solve continuous, not discrete, MDP

– Potential to overcome fine angular discretization problem – More representative of real world

• Transition associative array processing to aviation safety

assessments

– More advanced safety metrics possible since entire simulation can now be processed – Identify specific state combinations that logics are robust or susceptible to

Future Work

M.S Thesis Defense- 80 AJW 12/1/14

Andrew Weinert Assistant Technical Staff Surveillance Systems Email: andrew.weinert@ll.mit.edu Phone: (781) 981-0986

Acknowledgments

• Dr. Michael Owen*
• Dr. Mykel Kochenderfer

Robert Klaus Richard Williams Vijay Gadepally Randal Guendel Christine Parry Chansup Byun Gregory Hogan Gregg Shoults

• Dr. Paul Breimyer

MIT Lincoln Laboratory

• Dr. David Castañón*
• Dr. Ioannis Paschalidis*
• Dr. John Baillieul
• Dr. Mario Cabodi
• Dr. Michael Caramanis
• Dr. Prakash Ishwar
• Dr. Eric Schwartz
• Dr. Ari Trachtenberg

Austin Alexander Cali Ann Stephens

Boston University

*M.S Thesis Committee

M.S Thesis Defense- 81 AJW 12/1/14

• Enhanced TCAS II Algorithm
• Angular Rate States
• Simulation States
• Simulation Processing Details
• State-Transition Gen. Details
• Policy Generation with Noise
• Policies with Horizontal

Maneuvers

• Baseline SOC

Supplemental Slides Index

• Performance Metrics
• Encounter Model Categories
• Bayesian Networks
• Encounter Model Overview
• Coordination
• ACAS X Input
• ACAX X Tuning
• ACAS X Optimization
• ACAS X Feedback
• ACAS X Usage
• ACAS X Modifications

M.S Thesis Defense- 82 AJW 12/1/14

Enhanced TCAS II Algorithm

Bendix Corporation Prototype 𝑻 = 𝒔𝒊, 𝑻𝒑, 𝐓𝐣, 𝝉𝐄, 𝑻𝑺𝑩

State Name

𝒔𝒊 Straight line range 𝝉𝑬 Horizontal miss estimate 𝑻𝒑 Ownship speed 𝑻𝒋 Intruder speed

State Name

𝝉𝑪 Angular rate SD 𝝊𝒊 Horizontal tau 𝒕𝑺𝑩 Ownship RA state

Primary States Secondary States

𝑻𝒋 𝑻𝒑 𝒔𝒊 = 𝒕𝒑 + 𝒕𝒋 𝝊𝒊 𝟒𝝉𝑬

NMAC

𝝉𝑬 = 𝒕𝒑 + 𝒕𝒋 𝝊𝒊

𝟑𝝉𝑪

𝒕𝑺𝑩

Ownship Intruder

M.S Thesis Defense- 83 AJW 12/1/14

Angular Rate States

𝜠𝝎 = 𝟏° 𝝎𝑺 = 𝟏° 𝜷 = −𝟐𝟗𝟏° 𝜠𝝎 = 𝟏° 𝝎𝑺 = 𝟏° 𝜷 = 𝟏° 𝜠𝝎 = 𝟏° 𝝎𝑺 = 𝟏° 𝜷 = 𝟘𝟏° 𝜠𝝎 = 𝟏° 𝝎𝑺 = 𝟏° 𝜷 = −𝟘𝟏°

Difficult to represent encounter type with relative angular states. Bearing (𝜷) is advantageous because it quantifies the quadrant

𝝎𝟏 = 𝑫° 𝝎𝒋 = 𝑫°

𝜠𝝎 = relative heading 𝝎𝑺 = angle of resultant 𝜷 = bearing

M.S Thesis Defense- 84 AJW 12/1/14

• Airspeed
• Airspeed acceleration
• Altitude
• Altitude rate of change
• Bank angle
• Body-fixed angular vector
• Commanded vertical rate
• Command turn rate
• Commanded longitude

acceleration

• East position

Simulation States

• East position Rate
• Heading angle
• North position
• North position rate
• Pitch angle
• Pitch rate
• Turn rate
• Vertical acceleration
• Vertical rate
• Yaw angle
• Yaw rate

M.S Thesis Defense- 85 AJW 12/1/14

1. Store raw simulation result in 13,404 files with ≈10,000 lines per file

– File limit required for load balancing – Raw data files ranged in size from 0.18—2.56 Megabytes

2. Process raw files into associative arrays for D4M

– An associative array is represented by a triple store of row keys, column keys, and a value – Stored in MATLAB .mat file and require from 0.09—1.2 Megabytes – Row key concentration of encounter set id and Monte Carlo id – Column key concentration of time, state name, and state value – Value key is always one

Simulation Processing

Implementation Details

Simulation produced ≈ 6,970,000,000 numeric doubles Largest individual aircraft simulation data set ever generated

M.S Thesis Defense- 86 AJW 12/1/14

1. Generated a matrix containing all unique state combinations

– Structure is provided by sorting rows in ascending order by state – A kd-tree is generated for each individual state as well

2. Preallocate state-transition matrix as an empty sparse matrix.

– A linear index matrix that records every nonzero element in the state-transition is also preallocated

3. Filter relevant states using D4M and the nearest-neighbor discretization point for each simulation state are determined

– Index into state-transition using nearest-neighbor indices – Searching many single state kd-trees is faster than a large kd-tree

4. Update state-transition matrix

– For UAS set, filter out observations with vertical rates greater than representative UAS vertical rates

Calculate State-Transition

Implementation Details

M.S Thesis Defense- 87 AJW 12/1/14

• Optimization with observed dynamics can be noisy

– Resulting policy may not be completely smooth – Smoothing that doesn’t change the character of the policy required

• Mitigated through state-transition matrix smoothing or minimal

dilation / erosion of policy

Policy Generation

Simulation Noise

“Holes in the policy”

M.S Thesis Defense- 88 AJW 12/1/14

Policies with Horizontal Maneuvers

• Without angular information,

little incentive to select horizontal maneuvers

– Horizontal cost needs to be significantly less than the vertical cost to encourage horizontal maneuvers

• Inclusion on angular

information can produce similar policies

• Horizontal maneuvers “grow
• ut” from the NMAC region

when changing the cost

M.S Thesis Defense- 89 AJW 12/1/14

• A system operating characteristic (SOC) curve illustrates the

relationship between safety and operational components

• Since relationship between vertical and horizontal alert costs

was unknown, randomly assigned alert costs during

• ptimization and evaluated policy

Baseline 𝚬𝐢, 𝐬𝐢

Evaluation: SOC Curve

20 40 60 80 100 120 140 160 180 200 220 240 0.1 0.3

Alert Duration (s) NMAC Risk Ratio

0.2

Equal alert cost Random alert cost

M.S Thesis Defense- 90 AJW 12/1/14

Example Metrics Meets/Exceeds Requirements Risk ratio (unresolved, induced) ✓ Overall alert rate ✓ Corrective alert rate ✓ Disruptive alerts in normal operations (500’, 1,000’, parallels, 3NM) ✓ Location distribution (airspace, airports, altitude) ✓ Operation distribution (air carriers vs. business jets) ✓ Altitude change for corrective alerts ✓ Crossing alert rate ✓ Reversal rate ✓ Strengthening rate ✓ Magnitude of vertical rate change ✓ Yo-yo advisory sequences ✓ Alert timing, duration, termination ✓ Complex alert sequences ✓ And more… … Safety Operational Suitability Acceptability Operational Validation Plan Performance metrics, definitions, and required exit criteria

Performance Metrics

M.S Thesis Defense- 91 AJW 12/1/14

Encounter Model Categories

• Correlated

(cooperative)

– Prior U.S. model needed to be updated, captures RVSM – Assumes ATC involvement

• Uncorrelated 1200-code

(noncooperative surrogate)

– First model to capture encounters between VFR aircraft – Assumes no ATC involvement

• Uncorrelated

(unconventional aircraft)

– Models vehicles unlikely to carry transponders – Assumes no ATC involvement

Discrete code 1200/VFR Discrete code 1200/VFR Noncooperative Conventional Noncooperative Unconventional

Aircraft of interest Intruder aircraft

Conventional: General Aviation typical of 1200-code aircraft Unconventional: balloons, gliders, ultralights,…

MIT LL Models

Encounter model describe the nominal encounter situation without the SS/CA system

M.S Thesis Defense- 92 AJW 12/1/14

Conditional Probability Table P(Airspeed | Airspace, Altitude Layer) …

Bayesian Networks

• Bayesian networks graphically represent a set of

random variables and their conditional dependencies

• Nodes represent variables, arcs symbolize

dependencies

• Count each observed occurrence to build

probability tables

• Used to specify encounter initial conditions

M.S Thesis Defense- 93 AJW 12/1/14

Model Development Overview

Radar tracker Radar tracker Radar tracker Encounter Database Airspace Statistics Fusion tracker

Uncorrelated Model Correlated Model

134 radar sites ~15+ GB per day Models nominal VFR flight ~100,000 VFR flight hours 6 variables Models encounters between two aircraft ~800,000 encounters; 16 variables

VFR Track Database Feature Extraction Feature Extraction GPS tracker GPS Post- Processing

Unconventional Model

Feature Extraction

Models unconventional aircraft—e.g., paragliders, balloons, skydivers ~100,000 flight hours 5 variables

i.e., Class D airspace 1200 ft AGL 3 deg/s turn 1500 ft/min climb 80 kt airspeed 0 kt/s acceleration

M.S Thesis Defense- 94 AJW 12/1/14

• Model reports publically available
• Data tables, software for sampling models,

and density model available upon request

Documented in Academic Journals and Conferences Briefed to and Leveraged by Standards Organizations

AIAA Journal of Guidance, Control, and Dynamics March 2010 USA/Europe Air Traffic Management Research and Development Seminar July 2009 Lincoln Laboratory Journal December 2008 8th Integrated Communications, Navigation and Surveillance Conference May 2008 AIAA Guidance, Navigation, and Control Conference and Exhibit August 2008 AUVSI Unmanned Systems North America August 2010 RTCA SC-203 (Unmanned Aircraft/SAA) NATO Flight in Non-Segregated Airspace (FINAS) FAA UAS SAA Workshop

Used by Industry, Government, R&D Centers, Academia

Industry NGC, Honeywell, General Atomics, AeroVironment Utopia Compression, Bihrle, MTSI, SELEX Galileo Government/ R&D Centers AFRL, SIMAF, MITRE, JHU/APL, EDA MIDCAS Academia UND, MIT

Airspace Model Publications, Briefings, and Users

M.S Thesis Defense- 95 AJW 12/1/14

• Slightly different sensor measurements can result in both aircraft

selecting the same direction

• Coordination messages are sent to instruct other aircraft to not select

the same direction

• In case of a maneuver conflict, master aircraft overrides
• The following scenarios are to be assessed when evaluating

coordination:

– Both aircraft respond – Cross link disabled, both respond – Only own aircraft responds, but cross link is still enabled – All of the above with switching slave/master

Collision Avoidance Coordination Overview

Messages sent by Mode S interrogation/reply (currently) or ADS-B broadcast (future)

M.S Thesis Defense- 96 AJW 12/1/14

Meets design goals?

Human Expertise Computer Optimization Evaluation

Logic Requirements Tuning Parameters

Feedback and Tuning Process

Considerations Example Metric Performance Goals Safety

• Safer than TCAS

Operational Suitability

• Fewer alerts and less

disruptive than TCAS User Acceptability

• Pilots accept alerts

(or lack thereof)

Risk Ratio = Pr(NMAC with system) Pr(NMAC without system)

“Climb, Climb… Descend, Descend NOW… Climb, Climb NOW” Advisory Sequences

~500 feet Level-off Level

Alert frequency, rates, and types in standard visual encounters

ACAS X Methodology

Expert Input

M.S Thesis Defense- 97 AJW 12/1/14

Meets design goals?

Human Expertise Computer Optimization Evaluation

Logic Requirements Tuning Parameters

Feedback and Tuning Process

• σddh: nominal vertical acceleration
• dhmax: max achievable vertical speed
• E[Tinit]: mean initial pilot response delay
• E[Tsubs]: mean subsequent pilot response
• …

Offline Cost Function

Influences alerting behavior through computer

• ptimization

Influences alerting based on

• perational context
• Cconflict: vertical conflict
• Calert: initial alert
• Creversal: reversal
• Cclear of coflict: terminate alert (reward)

Online Parameters

• Cswitch: cost of switching advisory
• Crestart: cost of restarting advisory
• Cinhibit: cost of issuing descends

below certain altitudes

ACAS X Methodology

Tunable Parameters

M.S Thesis Defense- 98 AJW 12/1/14

Meets design goals?

Human Expertise Computer Optimization Evaluation

Requirements

Feedback and Tuning Process

Tuning Parameters Logic

Look-Up Table

0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 0 0 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 1 1 1 0 0 1 1 1 0 1 1 1 0 1 1 0 1 1 0 1 0 1 0 0 1 0 0 0 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 1 0 0 1 0 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 1 0 1 0 0 0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 0 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 0

Optimization Algorithm Offline Costs Probabilistic Dynamic Model Alert Types Online Parameters

(and simulation framework)

ACAS X Methodology

Logic Optimization

M.S Thesis Defense- 99 AJW 12/1/14

Meets design goals?

Human Expertise Computer Optimization Evaluation

Requirements Tuning Parameters Logic

Feedback and Tuning Process

Evaluation Results vs. Metrics Feedback for Additional Tuning

  

Risk ratio is better than TCAS



Issues too many disruptive alerts Alert sequences are not acceptable Does not alert in visual procedures

Most Wanted List 1) 2) 3) 4) 5)

Desired behavior provided to team for further tuning

8+ logic evaluations since June 2011

ACAS X Methodology Feedback and Tuning Process

M.S Thesis Defense- 100 AJW 12/1/14

ACAS X

Real-Time Logic Usage

Sensor Measurements State Distribution

Action Selection

Resolution Advisory Optimized Logic Table

0 0 0 0 0 1 0 1 0 1 0 1 1 0 0 0 0 0 0 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 0 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 1 1 1 0 0 1 1 1 0 1 1 1 0 1 1 0 1 1 0 1 0 1 0 0 1 0 0 0 0 1 1 0 0 1 0 1 1 1 1 0 1 1 1 1 1 0 1 1 0 0 1 0 1 1 1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 1 0 1 0 0 0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 0 0 0 1 0 1 1 1 0 0 1 0 0 0 0 1 1 1 1 0 0 0 1 1 0

Updates

• nce per second

State Estimation

Probabilistic Dynamic Model Probabilistic Sensor Model

Fast table lookups

M.S Thesis Defense- 101 AJW 12/1/14

Surveillance Model Parameters Rules TCAS

Modify tracker and logic assumptions Modify assumed delay and strength Modify alert thresholds Modify pseudocode

ACAS X

Modify sensor model Modify transition probabilities Modify event costs Modify

• nline costs

IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT

Tau = 40 s Zthr = 600 ft ALIM = 300 ft NMAC = -1 Alert = -0.01 Reversal = -0.01 Alert inhibit altitude

IF (ITF.A LT G.ZTHR) THEN IF(ABS(ITF.VMD) LT G.ZTHR) THEN SET ZHIT; ELSE CLEAR ZHIT; ELSE IF (ITF.ADOT GE P.ZDTHR) THEN CLEAR ZHIT

System Modifications

Both TCAS and ACAS X selected system states by hand and have not quantified the utility of each state

View publication stats View publication stats