Definition

Safety performance monitoring and measurement is one of three elements that comprise the safety assurance component of the International Civil Aviation Organisation (ICAO) safety management system (SMS) framework. Safety assurance consists of processes and activities undertaken by the service provider to determine whether the SMS is operating according to expectations and requirements.

Safety performance monitoring and measurement represents the means to verify the safety performance of the organisation and to validate the effectiveness of safety risk controls. (ICAO Doc 9859 - Safety Management Manual)

EUROCONTROL has defined it as a systematic action conducted to detect changes affecting the air traffic management (ATM) system with the specific objective of identifying that acceptable or tolerable safety can be met. (EUROCONTROL ESARR3)

Description

Regulatory requirements related to safety management and implementation of an SMS mandate aviation service providers to develop and maintain the means to verify the safety performance of the organisation. Safety performance monitoring and measurement enables continuous monitoring and regular assessment of the safety levels achieved by an organisation during service delivery. The safety performance shall be verified in reference to the safety policy and approved safety objectives (safety performance targets) using dedicated safety performance indicators (SPIs). Trends should be analysed and, consequently, corrective actions should be identified after detecting any deterioration of specified safety levels.

Safety monitoring could be an important source of information for measuring the effective functioning of other SMS components. According to EUROCONTROL ESARR4 - Risk Assessment and Mitigation in ATM, safety monitoring and data collection mechanisms could be specifically developed and used for the validation of the safety assumptions and requirements identified in the risk assessment and mitigation processes, as well as for the validation of safety models used in risk management. Also, safety monitoring and data collection support the development of safety indicators at national, regional and global level.

Information Sources for Safety Monitoring

Information used to measure an organisastion’s safety performance is generated through its safety reporting systems, and there are two types of reporting systems:

• Mandatory incident or occurrence reporting systems, which require the reporting of certain types of events, such as serious incidents and accidents; and
• Voluntary incident or occurrence reporting systems, which allow for the submission of information related to observed hazards or inadvertent errors without a legal requirement to do so.

Other sources of safety information to support safety performance monitoring and measurement may include:

• Safety audits and informal inspections, which focus on the integrity of the organisation's SMS and its supporting systems.
• Safety surveys that examine procedures or processes related to a specific operation.
• Safety occurrence reporting
• Investigation of safety occurrences
• Safety studies, which are rather large analyses encompassing broad safety concerns. An in-house safety analysis can find arguments and make use of findings and recommendations of studies carried out at industry-wide or global scale. Because of their nature, safety studies are more appropriate to address system safety deficiencies rather than identify specific, individual hazards.
• Safety reviews that are conducted during introduction and deployment of new technologies, change or implementation of procedures, or in situations of a structural change in operations. Safety reviews are a fundamental component of the management of change. They have a clearly defined objective that is linked to the change under consideration.
• Systematic capture of data reflecting actual day-to-day performance (by means of dedicated programmes, such as Flight Data Monitoring, Line Operations Safety Audit (LOSA) and Normal Operations Safety Survey (NOSS))
• Continuous monitoring of day-to-day activities related to the delivery of services.

Each of these types of information source may be available to an organisation. However, specifications about what these sources should be and what they should “look like” are not available - individual organisations should tailor them to the appropriate scope and scale.

Procedures and Arrangements for Safety Monitoring

The size and structure of the organisation, and the operational environment have to be considered when setting up safety monitoring arrangements in the organisation. The guidance material developed by EUROCONTROL suggests the following basic steps for establishing effective safety monitoring and measurement processes:

• Identification of indicators to be monitored

The scope of monitoring should cover operational, technical and organisational (safety management) aspects. Safety indicators can be quantitative or qualitative, leading (proactive) or lagging (reactive).

• Collation of the information for safety monitoring

There should be a systematic collation and evaluation of results from all safety monitoring activities to ensure that interrelationships can be detected.

• Analysis of indicators

The evolution of the indicators should be analysed, trends and related causes and influencing factors established.

• Application of corrective action process

Corrective actions should be determined, taken and followed up wherever the monitoring shows that an element is approaching a point which may affect safety to an unacceptable extent; coordination with relevant units/organisations should take place wherever necessary; the indicators and their evolution should be documented as well as actions taken and their results.

Safety Monitoring Methods and Programmes

Not long ago the safety monitoring processes relied mainly on staff identifying existing or potential risks and hazards by using their best judgement, experience and skills. Often if unsafe practices are used regularly as a normal method of operation, it is very likely that personnel involved in these activities would not recognise the actual and potential threats.

The need to improve the control of human error and develop countermeasures to error in operational environments lead to the introduction of observation-based methods, such as LOSA and NOSS and data analysis programmes, such as FDA. These methods are used to systematically capture data which reflect actual day-to-day performance and provide proactive means for safety monitoring and improvement.

Related Articles

• Flight Data Monitoring
• Line Operations Safety Audit (LOSA)
• Normal Operations Safety Survey (NOSS)

Further Reading

ICAO

• ICAO Doc 9859 - Safety Management Manual, Fourth Edition - 2018;
• ICAO Integrated Safety Management website;

European Commission

• Regulation 2017/373 Requirements for Providers of ATM/ANS and other ATM Network Functions and their Oversight;

EUROCONTROL

• Eurocontrol Safety Regulatory Requirement - ESARR3, Use of Safety Management Systems by ATM Service Providers;
• ESARR Advisory Material/Guidance Document - EAM3/GUI3 ESARR3 and Related Safety Oversight;
• Safety Regulatory Requirement - ESARR 4, Risk Assessment and Mitigation in ATM.
• Safety KPIs presentation, October 2010

FAA

• Air Traffic Organization 2012 Safety Report
• Safety Performance Metric FAA, 2010

Flight Safety Foundation

• Year Three Report of the Global Safety Information Project, Flight Safety Foundation, 2017.
• Safety Performance Monitoring Survey results dashboard, 2018.