Integrated Approach of Risk-based Compliance-based - - PowerPoint PPT Presentation

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Integrated Approach of Risk-based Compliance-based Performance-based Audit and Surveillance Programme

FSSCO, Busan, 29 November 2017

George WAI, Airworthiness Officer Flight Standards and Airworthiness Division Hong Kong Civil Aviation Department

1

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Outline

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

• Purpose of the integrated approach
• Methodology
• Result

2

3

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

The Purpose

Enhance the prioritisation in the Audit and Surveillance Programme:

• Introduce a Data-driven, Risk-based approach
• Integrate it with the existing Compliance-based

and Performance-based approaches

4

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Integrated Approach

5

Risk-based: Risk assessment, areas of importance

Compliance-based: Administrative procedures, rules Performance-based: Measureable

• utcomes, not

prescriptive

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

The Guidance

ICAO Doc 9859 Safety Management Manual 3rd Edition

6

7

AOC Annual Report Accident Report Serious Incident Voluntary Report Biannual Report Extended Diversion Time Operations Audit Report Safety Assessment Foreign Aircraft

Data to be considered

Mandatory Occurrence Report Requirements Survey Reliability

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Suitable Safety Performance Indicators (SPI)

8

LOW

Audit findings Aircraft survey Programme

HIGH HIGH

MOR EDTO/IFSD MOR EDTO/IFSD

Safety Data

Note: IFSD (Inflight Shut Down)

• Based on Data-driven
• Use of existing safety oversight activities
• Consolidated and categorised

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

AOC A1 A2

Scheduled Flight Unscheduled Flight

MRO M1 M2

Base / Engine Maintenance Line Maintenance Only

9

H

Component Avionics Overhaul NDT Plating

RISK GROUP

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Consequence Indicators

• Monthly Monitoring

10

LOW Audit findings Aircraft Survey Programme (ASP) HIGH MOR EDTO/IFSD

• Annual Monitoring

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Risk Group High Consequence Indicators Low Consequence Indicators MOR EDTO-IFSD Audit Finding Aircraft Survey A1 √ √ √ √ A2 √ √ M1 √ √ M2 √ H √

11

Allocation of SPIs to Risk Groups

(1) MOR & IFSD SPI (3) Audit & ASP SPI (2) MOR SPI (4) Audit SPI (5) ASP SPI

• Weighed cognitively
• Allocated based on complexity
• Combined to form 5 SPIs

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

12

MOR & IFSD SPI (Risk Group A1)

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Risk Group High Consequence Indicators Low Consequence Indicators MOR EDTO/IFSD Audit Finding Aircraft Survey A1 √ √ √ √ A2 √ √ M1 √ √ M2 √ H √

13

MOR SPI (Risk Group A2 & M1)

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Risk Group High Consequence Indicators Low Consequence Indicators MOR EDTO/IFSD Audit Finding Aircraft Survey A1 √ √ √ √ A2 √ √ M1 √ √ M2 √ H √

14

Audit & Aircraft Survey SPI (Risk Group A1)

• Level 1 Finding: Significant non-compliance lowers the safety standard and

hazards seriously the flight safety.

• Level 2 Finding: Any non-compliance could lower the safety standard and

possibly hazard the flight safety.

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

15

Audit SPI (Risk Groups A2, M1, M2, H)

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

16

Aircraft Survey (Risk Group A1)

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Target / Alert Level

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

• Made reference to

ICAO Doc 9859

• Established 5

zones with past 3 years data

17

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Example: High-consequence Indicator

Occurrence (MOR & EDTO/IFSD) & No. of Aircraft

18

Organisation 2013 2014 2015 2016 2013 2014 2015 2016 Operator-MOR No Risk (RF 1) 10 15 12 4 41 41 42 42 Low Risk (RF 2 &3) 13 45 62 31 41 41 42 42 Medium Risk (4 & 5) 4 1 41 41 42 42 High Risk (6 & 7) 41 41 42 42 EDTO IFSD 2 41 41 42 42 Total No. of Occurrence Total No. of Aircraft

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Simple Mathematics

19

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

20

Organisation 2013 2014 2015 2016 2013 2014 2015 2016 Operator-MOR Risk % No Risk (RF 1) 0.24 0.37 0.29 0.10 5% 0.01 0.02 0.01 0.00 Low Risk (RF 2 &3) 0.32 1.10 1.48 0.74 10% 0.03 0.11 0.15 0.07 Medium Risk (4 & 5) 0.10 0.00 0.02 0.00 20% 0.02 0.00 0.00 0.00 High Risk (6 & 7) 0.00 0.00 0.00 0.00 35% 0.00 0.00 0.00 0.00 EDTO IFSD 0.00 0.00 0.00 0.05 30% 0.00 0.00 0.00 0.01 0.06 0.13 0.17 0.09 Average - 0.5 SD Lower Target Threshold 0.09 0.09 0.09 0.09 Average 0.11 0.11 0.11 0.11 Average + 0.5 SD Upper Target Threshold 0.13 0.13 0.13 0.13 Average + 1 SD 0.16 0.16 0.16 0.16 Average + 2 SD 0.20 0.20 0.20 0.20 Occurrence Rate (per aircraft) SPI - Occurrence Rate Overall SPI Target 1st Alert Level 2nd Alert Level

Occurrence Rate (EDTO/IFSD) and SPI Establishment

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

SPI for Occurrence Rate (MOR & EDTO/IFSD)

21

(1) MOR & IFSD SPI

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Audit Frequency Adjustment

22

Dominating Factor

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Audit Frequency Adjustment

Region Individual SPI Combined SPI (Dominate SPI Dependent) Overseas Organisation Risk Group(M1, M2 and H) G1

• 5%
• 5% or -10%

Unchanged G2 Unchanged Unchanged Unchanged R1 +5% +5% or +10% Unchanged R2 +10% +10% or +15% Unchanged R3 +15% +15% +1 Audit

• Provided expected audit frequency adjustment based on the integrated

approach

• Overseas M1, M2, H organisations: Audit arranged once per year/two-year
• cycle. Influenced by small variation by number of finding

23

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Implementation

• Held internal meetings
• Discussed comments
• Refined this methodology
• Established 2018 audit &

surveillance programme

• Review SPI target and alert levels

annually

24

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

Conclusion

Civil Aviation Department The Government of the Hong Kong Special Administrative Region

25

• Use risk-based approach on oversights
• Divide AOC/MRO into 5 Risk Groups
• Find suitable high/low consequence indicators
• Prioritise oversight based on analysis results

Civil Aviation Department The Government of the Hong Kong Special Administrative Region Civil Aviation Department The Government of the Hong Kong Special Administrative Region

26

YOU CAN’T HAVE

A RAINBOW

WITHOUT THE RAIN

End of Presentation