Metrics and Performance Metrics and Performance Management - - PowerPoint PPT Presentation

▶
metrics and performance metrics and performance
SMART_READER_LITE
LIVE PREVIEW

Metrics and Performance Metrics and Performance Management - - PowerPoint PPT Presentation

Sep 16, 2022 279 likes •499 views

Metrics and Performance Metrics and Performance Management Management W. Post W. Post Wednesday, 20 July 2005 Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar 6th USA/Europe ATM R&D Seminar - - Baltimore, MD Baltimore, MD 1

slide-1
SLIDE 1

Wednesday, 20 July 2005 Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar 6th USA/Europe ATM R&D Seminar -

  • Baltimore, MD

Baltimore, MD 1 1

Metrics and Performance Metrics and Performance Management Management

  • W. Post
  • W. Post
slide-2
SLIDE 2

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 2

General objective General objective

  • create and reinforce

create and reinforce working and personal working and personal relationships relationships between leading experts and between leading experts and researchers in the ATM R&D community, researchers in the ATM R&D community, share available results share available results and and reach consensus reach consensus

  • n major issues
  • n major issues. Since the goal of these

. Since the goal of these seminars is to seminars is to “ “create and reinforce create and reinforce” ”, an , an emphasis in the selection process will be on emphasis in the selection process will be on identifying and identifying and tracking continuity and tracking continuity and progress progress from previous seminars. from previous seminars.

slide-3
SLIDE 3

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 3

Track scope Track scope

  • "ATM performance assessment, modeling

"ATM performance assessment, modeling and analysis techniques, gate and analysis techniques, gate-

  • to

to-

  • gate (en

gate (en-

  • route to en

route to en-

  • route) capacity issues, demand

route) capacity issues, demand forecasting, demand management; this also forecasting, demand management; this also includes wider ATM business related includes wider ATM business related issues." issues."

slide-4
SLIDE 4

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 4

Layered Performance Layered Performance Management in ATM Management in ATM

= Impact relationship

Aviation & Society

Level 1 ATM systems RASP Level 2 ATM core services RTSP Level 3 sub-functions, subsystems & enablers RNP, RCP, … Level 4 Technologies and methods SARPS, … Level 5 ATM Community Expectations KPAs KPAs KPAs KPAs KPAs

Static aggregates Dynamics

slide-5
SLIDE 5

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 5

Key Performance Areas Key Performance Areas

KPA 01 Access and Equity KPA 02 Capacity KPA 03 Cost Effectiveness KPA 04 Efficiency KPA 05 Environment KPA 06 Flexibility KPA 07 Global Interoperability KPA 08 Participation by the ATM community KPA 09 Predictability KPA 10 Safety KPA 11 Security Note1: Mostly applicable to upper 2 to 3 layers. Different KPAs could be applicable to lower layers

slide-6
SLIDE 6

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 6

Coverage of track papers Coverage of track papers

Level Level 1 1 2 2 3 3 4 4 5 5 KPA 01 Access and Equity KPA 01 Access and Equity KPA 02 Capacity KPA 02 Capacity KPA 03 Cost Effectiveness KPA 03 Cost Effectiveness KPA 04 Efficiency KPA 04 Efficiency KPA 05 Environment KPA 05 Environment KPA 06 Flexibility KPA 06 Flexibility KPA 07 Global Interoperability KPA 07 Global Interoperability KPA 08 ATM community Participation KPA 08 ATM community Participation KPA 09 Predictability KPA 09 Predictability KPA 10 Safety KPA 10 Safety KPA 11 Security KPA 11 Security

slide-7
SLIDE 7

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 7

Observations Observations

  • Common set of

Common set of KPIs KPIs for reporting at ATM system for reporting at ATM system level (2) is still missing. level (2) is still missing.

  • Causality between the studied performance at

Causality between the studied performance at lower level and desired change at system level is lower level and desired change at system level is usually implicit. usually implicit.

  • Are we focussing on the 1

Are we focussing on the 1st

st, 2

, 2nd

nd, or 3

, or 3rd

rd order

  • rder

effects? effects?

  • Studies are often specialised to a particular local

Studies are often specialised to a particular local problem. problem.

  • Data availability still often an issue.

Data availability still often an issue.

slide-8
SLIDE 8

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 8

Conclusions Conclusions

  • Main focus of papers

Main focus of papers in track in track is on few is on few performance areas: performance areas:

  • Capacity, Efficiency, Cost effectiveness,

Capacity, Efficiency, Cost effectiveness, Environment, Safety (papers from SAF track not Environment, Safety (papers from SAF track not counted). counted).

  • Some other areas are addressed in more detail

Some other areas are addressed in more detail in other tracks (e.g. safety, environment). in other tracks (e.g. safety, environment).

  • Value of ATM interesting 'new' topic.

Value of ATM interesting 'new' topic.

slide-9
SLIDE 9

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 9

Conclusions Conclusions

  • Need for a commonly applied ATM performance

Need for a commonly applied ATM performance framework is high. framework is high.

  • R&D,

R&D,

  • Daily performance assessment,

Daily performance assessment,

  • Target setting.

Target setting.

  • Standardise metrics definitions.

Standardise metrics definitions.

  • How to deal with trade

How to deal with trade-

  • offs at all levels?
  • ffs at all levels?
  • Need to

Need to build build common understanding common understanding of current

  • f current

and future (desired) ATM performance. and future (desired) ATM performance.

slide-10
SLIDE 10

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 10

Further details Further details

  • Paper summaries can be found in

Paper summaries can be found in subsequent slides of this presentation but subsequent slides of this presentation but are not presented. are not presented.

  • Some slides with definitions from ICAO ATM

Some slides with definitions from ICAO ATM Performance Manual work are provided but Performance Manual work are provided but not presented. not presented.

slide-11
SLIDE 11

Wednesday, 20 July 2005 Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar 6th USA/Europe ATM R&D Seminar -

  • Baltimore, MD

Baltimore, MD 11 11

Questions? Questions?

slide-12
SLIDE 12

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 12

Session 1 Session 1

  • Economic Contribution and Productivity of ATM

Economic Contribution and Productivity of ATM -

  • Dr. Paula Leal de Matos
  • Dr. Paula Leal de Matos –

– Eurocontrol Eurocontrol.

.

  • KPA 03

KPA 03 – – level 1. level 1.

  • Amongst first studies to address ATM Added value

Amongst first studies to address ATM Added value

  • Difficulty to collect required economic data.

Difficulty to collect required economic data.

  • North America and Europe: 22 billion US$,

North America and Europe: 22 billion US$, 230.000 jobs. 230.000 jobs.

  • Productivity comparable to other technologically

Productivity comparable to other technologically advanced sectors. advanced sectors.

  • True value of ATM may need to be further clarified

True value of ATM may need to be further clarified

slide-13
SLIDE 13

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 13

Session 1 Session 1

  • Extent and Impact of Future NAS Capacity

Extent and Impact of Future NAS Capacity Shortfalls in the United States: A Socio Shortfalls in the United States: A Socio-

  • Economic Demand Study

Economic Demand Study -

  • Shahab

Shahab Hasan Hasan – – LMI/David Ballard LMI/David Ballard – – GRA, Inc. GRA, Inc.

  • KPA 03

KPA 03 – – level 1. level 1.

  • FAA Long term forecast & Capacity

FAA Long term forecast & Capacity constraints from 2012. constraints from 2012.

  • Lack of capacity would cost from 6.5 (2015)

Lack of capacity would cost from 6.5 (2015) to 19.5 billion US$. Cumulative between 90 to 19.5 billion US$. Cumulative between 90 and 230 billion US$ for period 2015 and 230 billion US$ for period 2015-

  • 2025.

2025.

slide-14
SLIDE 14

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 14

Session 2 Session 2

  • Flight Efficiency Studies in Europe and the

Flight Efficiency Studies in Europe and the United States United States -

  • Tarja Kettunen

Tarja Kettunen – – ISA Software/Dave ISA Software/Dave Knorr Knorr – – FAA FAA

  • KPA 04

KPA 04 – – Level Level 2. 2.

  • Indicator:

Indicator: Excess distance / direct distance. Excess distance / direct distance.

  • Radar data versus great circle route.

Radar data versus great circle route.

  • Inefficiency:10%, distance and city

Inefficiency:10%, distance and city-

  • pair

pair

  • dependant. Extra Cost 1 billion
  • dependant. Extra Cost 1 billion €

€ /year. /year.

  • Causes: Route structure, TMA Ops, Mil. Areas.

Causes: Route structure, TMA Ops, Mil. Areas.

  • 70% TMA, 30% en route. Same for US en EU.

70% TMA, 30% en route. Same for US en EU.

  • Mature Indicator. Need to include vertical profile

Mature Indicator. Need to include vertical profile

slide-15
SLIDE 15

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 15

Session 2 Session 2

  • A Preliminary Analysis of the Impact of Mile

A Preliminary Analysis of the Impact of Mile-

  • In

In-

  • Trail Restrictions on NAS Flight

Trail Restrictions on NAS Flight Operations Operations –

– Tim Myers

Tim Myers – – Metron Metron Aviation Aviation, Inc , Inc.

.

  • KPA 02 + 09

KPA 02 + 09 – – Level 2 + 3 Level 2 + 3

  • NTML and ETMS data are not integrated.

NTML and ETMS data are not integrated.

  • 32% of

32% of MITs MITs < 5 flights. Necessary? < 5 flights. Necessary?

  • Suggests no major impact on en route

Suggests no major impact on en route delay, actual en route spacing, or GDP delay, actual en route spacing, or GDP arrival compliance. arrival compliance.

slide-16
SLIDE 16

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 16

Session 3 Session 3

  • Modeling Delays and Cancellation

Modeling Delays and Cancellation Probabilities to Support Strategic Probabilities to Support Strategic Simulations Simulations -

  • David Lovell

David Lovell – – University of Maryland University of Maryland

  • KPA 02

KPA 02 – – Level 2 Level 2

  • Delays & Cancellations

Delays & Cancellations -

  • > Cost.

> Cost.

  • Simple & expedient models.

Simple & expedient models.

  • Strategic simulations & Slot Auctions.

Strategic simulations & Slot Auctions.

  • Low levels of airline proprietary information.

Low levels of airline proprietary information.

slide-17
SLIDE 17

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 17

Session 3 Session 3

  • Estimation of Arrival Capacity and Utilization at

Estimation of Arrival Capacity and Utilization at Major Airports Major Airports -

  • Husni

Husni Idris Idris / / Antony Antony Evans Evans – – Titan Corporation Titan Corporation

  • KPA 02

KPA 02 – – Level 2 + 3. Level 2 + 3.

  • Using historic data and modelling.

Using historic data and modelling.

  • Achievable arrival capacity & current utilisation.

Achievable arrival capacity & current utilisation.

  • Smaller

Smaller APTs APTs underutilised by lack of demand. underutilised by lack of demand.

  • No saturation.

No saturation.

  • BOS also underutilised by inefficient operations.

BOS also underutilised by inefficient operations.

  • Airport analysis of demand growth potential and/or

Airport analysis of demand growth potential and/or efficiency improvement potential. efficiency improvement potential.

slide-18
SLIDE 18

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 18

Session 3 Session 3

  • The Feasibility of Measuring Capacity in a Real

The Feasibility of Measuring Capacity in a Real-

  • Time ATM

Time ATM Simulation Independent of Subjective Controller Workload Simulation Independent of Subjective Controller Workload Measurement Measurement –

– Mick van Mick van Gool Gool –

– EUROCONTROL/ Kevin Corker EUROCONTROL/ Kevin Corker – – SJSU. SJSU.

  • KPA 02

KPA 02 – – Level 3 + 4 Level 3 + 4

  • Verify Integra output with Air MIDAS.

Verify Integra output with Air MIDAS.

  • HITL experiments processed by both methods.

HITL experiments processed by both methods.

  • Information Processing Load as Indicator.

Information Processing Load as Indicator.

  • Similar patterns yet higher values for Integra.

Similar patterns yet higher values for Integra.

  • Effect of Air MIDAS Scheduler?

Effect of Air MIDAS Scheduler?

slide-19
SLIDE 19

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 19

Terminology Terminology

KPA Expectation Performance Objective Performance Indicator Performance Target Performance Metric

slide-20
SLIDE 20

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 20

Terminology (1) Terminology (1)

  • Expectation

Expectation

  • 11 high level expectations of society are defined in the ICAO OC

11 high level expectations of society are defined in the ICAO OCD D (safety, capacity, cost effectiveness etc.). (safety, capacity, cost effectiveness etc.).

  • Key Performance Area (KPA)

Key Performance Area (KPA)

  • 11

11 KPAs KPAs, one per expectation as defined in the ICAO OCD. , one per expectation as defined in the ICAO OCD. KPAs KPAs named after their corresponding expectation. named after their corresponding expectation.

  • Used as categories throughout the performance framework, to

Used as categories throughout the performance framework, to group related group related performance objectives. performance objectives.

  • Performance Objective

Performance Objective

  • Each

Each expectation expectation should be reached through meeting a set of should be reached through meeting a set of specific, measurable, achievable, relevant and timely (SMART) specific, measurable, achievable, relevant and timely (SMART)

  • bjectives.
  • bjectives.
  • Objectives

Objectives define define – – in a qualitative way in a qualitative way -

  • a desired trend from

a desired trend from today today’ ’s performance ( s performance (eg eg improvement), within a well specified improvement), within a well specified ATM planning environment ( ATM planning environment (eg eg each objective is applicable within each objective is applicable within the scope of a given geographical area, time period and other the scope of a given geographical area, time period and other scope scope-

  • limiting criteria).

limiting criteria).

slide-21
SLIDE 21

Wednesday, 20 July 2005 6th USA/Europe ATM R&D Seminar - Baltimore, MD 21

Terminology (2) Terminology (2)

  • Performance Indicator

Performance Indicator

  • Indicators

Indicators are defined when there is a need to numerically are defined when there is a need to numerically document current performance levels and progress in achieving document current performance levels and progress in achieving an an objective.

  • bjective.
  • Performance Target

Performance Target

  • A set of agreed numerical values of related

A set of agreed numerical values of related performance performance indicators indicators, representing the minimum performance levels at which , representing the minimum performance levels at which an an objective

  • bjective is considered to be

is considered to be ‘ ‘achieved achieved’ ’. .

  • Performance Metric

Performance Metric

  • A generic definition of what can be measured, how it can be

A generic definition of what can be measured, how it can be measured and in which context and scope this should be done. measured and in which context and scope this should be done. Defines also the units in which the measurement is to be Defines also the units in which the measurement is to be expressed. expressed.