Assessing Safety Culture in ATM

Assessing Safety Culture in ATM

Article Information
Category:
Content source:
EUROCONTROL
Content control:
SKYbrary

Objective

The objective of an Air Navigation Service Provider Safety Culture assessment is to establish a widely shared understanding of the organisation’s Safety Culture and identify its strengths and weaknesses, in order to determine whether change is desirable and identify the way ahead.

Description of the Process

The Safety Culture assessment process entails the use of both qualitative and quantitative techniques to capture the nature and essence of the operational context under evaluation. The process involves:

  • a pre-launch phase to prepare the ANSP senior managers and personnel for the assessment and to obtain their support
  • data collection phases, involving both quantitative processes (questionnaire survey) and qualitative processes (interviews, workshops / focus groups, site visits, reviews of historical information and contextual data analysis, etc.), and, depending on time and accessibility,
  • Safety Culture analysis
  • Diagnosis, feedback and the way ahead.

Safety Culture Assessment Teams

A collaboration between ANSP members and an independent Safety Culture assessment team (compiled of external experts) is desirable for collecting pertinent information, analysing the findings, and drawing conclusions for planning and implementing improvements to the existing Safety Culture.

An external Safety Culture assessment team is useful for:

  • contributing an ‘outsider’ perspective. It is often easier for ‘outsiders’ rather than ‘insiders’ to identify subtle but important aspects of organisational culture, investigate the values and beliefs underlying day-to-day behaviours, appreciate past and present culture, and challenge individual and organisational safety behaviours
  • providing the expertise and experience necessary for conducting an effective Safety Culture assessment process, identifying Safety Culture strengths and weaknesses and working with ANSPs to investigate a way ahead.

The external expertise is frequently represented by an assessment team rather than a single expert to avoid, as far as possible, any bias or individual subjectivity.

The contribution of ANSP personnel is essential for:

  • providing the ‘inside’ information to measure the organisation’s Safety Culture (e.g., opinions and experiences on risk awareness, safety behaviour and commitment to and involvement in safety)
  • assisting the Safety Culture assessment team in data collection and validation (e.g., providing access to information and workshop participants)
  • establishing a viable way ahead
  • ensuring ‘ownership’ by the ANSP of the process, findings and outcomes.

Preparing for the Assessment

Prior to the launch of the assessment process, the ANSP and external assessment team should discuss and agree on details of the assessment process, such as:

Scope

The scope of involvement in the assessment process will be dependent on factors such as the project timeframe and the availability of ANSP personnel. At a minimum, the assessment should include representatives of the management group and operational staff from each site operated by the ANSP (e.g., Headquarters, regional centres, towers). Collecting a wide, representative range of views from all areas and levels of the ANSP is important in order to:

  • ensure that the assessment is an organisational Safety Culture assessment
  • enable the assessment team to compare and contrast the perceptions of different groups or sub-cultures, for example, to test whether managers do what they say they do, in the eyes of controllers
  • minimise any potential bias effects that may result when characteristics of an organisation are being assessed through a sample of individuals nominated to interact with the assessment team.

Timeframe

The assessment team should define, with the ANSP, an adequate time-period in which to launch the assessment

Launching the Assessment

The launch starts with a presentation of the entire process to senior and middle managers, representatives of operational staff (either separately or jointly), and other stakeholders if required (e.g., unions). Further presentations to separate working-groups can also be planned if necessary to motivate ANSP staff and ensure their active participation.

The kickoff presentations should introduce the assessment team to ANSP management and personnel and should cover:

  • the purpose and intended outcomes of the assessment
  • the process involved, including how information will be collected, analysed and used
  • the requirements of survey participants and the importance of their contributions
  • confidentiality terms. These must be explained to all participants who will be involved in the data gathering phase
  • any questions or concerns raised by ANSP personnel.

The presentation must capture the interest of the participants to ensure their commitment. It is an opportunity to ‘sell’ the project to those who will be giving their time to participate. It is thus important to promote the benefits to employees of a good Safety Culture, and to alleviate any concerns (‘threats’) they may perceive.

Data Collection: The Safety Culture Questionnaire

The Safety Culture questionnaire is a set of statements that respondents are asked to agree or disagree with by marking their choice on a five-point scale. The statements are divided into 4 sections; a general section for everyone, one section for controllers, another for technical staff (engineers, technicians and maintenance support) and a third set for managers. Two open-ended questions are included to collect further comments on the questionnaire itself and other relevant issues that personnel feel have not been covered by the questionnaire. The questionnaire is designed to elicit responses on a variety of topics that indicate how the ANSP approaches and manages safety in practice.

Distribution - The questionnaire can be distributed either by dissemination of hard-copies or by electronic means (e.g., the ANSP intranet, by notifying relevant personnel of the link to the electronic questionnaire). The questionnaire should only be completed once by each individual in the organisation and personnel should have access only to those sections that concern them (i.e., ATCOs should only have access to the section for ATCOs and not the section designed for maintenance technicians). Promotion and Sample Size - To ensure the validity of the questionnaire data it is important to have a representative sample size (at least 30% of the identified target groups, and ideally 50- 80%).

Therefore, the launch of the questionnaire should be combined with the familiarisation visits to the different units and locations (by the external assessment team) to explain the purpose of the survey and promote it. The ‘champion’ should also help to distribute the questionnaires and encourage people to complete and return them. Management support and encouragement are extremely valuable during this phase.

Collection - Once completed, the questionnaires are returned directly to the assessment team.

Data Collection: Familiarisation Visits

The main objectives of familiarisation visits are to:

  • provide the assessment team with a general impression of the climate and operational environment of the ANSP, and
  • enable the assessment team to interact with ANSP personnel during their day-to-day activities.

This process helps the assessment team to become familiar with the working environment of the ANSP and make observations that can be followed up and explored further in interviews and workshops. It also allows the team to establish a rapport with ANSP personnel, which is of general benefit in subsequent information sharing stages. In order to gather information on a range of Safety Culture issues, enablers, constraints and opportunities for improvement, the assessment team should visit different units of the ANSP such as towers, approach and en route centres.

Familiarisation visits are generally conducted by a mixed team, consisting of:

  • those with (ATC) operational expertise - to assist with understanding the technical aspects of the observed activity; and
  • external experts from the assessment team - who contribute an ‘outsider’ perspective and the experience and expertise to assess observed behaviours in terms of Safety Culture elements.

Data Collection: Additional Sources of Information

When assessing an ANSP’s Safety Culture, the assessment team must have an understanding of, and empathy with, the entire local environment in which the ANSP operates. As Hofstede notes, “ …people in other countries may think, feel, and act very differently from yourself, even when confronted with basic problems of society…”. The assessment team can conduct external literature and document reviews and websites reviews, for instance, to access relevant information. Reading about the historical background of the country may also assist in understanding the national culture. Specific information about an ANSP’s Safety Culture can be gained from a number of sources to complement data obtained through the questionnaires and familiarisation visits:

  • The ANSP website. The content and the way the information is presented conveys information about where the company’s areas of interest lie
  • Safety programs and safety initiatives in place within the ANSP, indicating the level of commitment to safety
  • Documented policies and procedures
  • Internal publications outlining details of the organisational structure, mission statement etc.
  • Incident reports, incident summary data and other safety indicators. While the Safety Culture questionnaire identifies whether the ANSP has tools in place to learn (e.g., reporting systems, incident analysis, feedback and communication channels), the quality and effectiveness of this learning process is difficult to assess without observing the outcomes. An examination of the process and related documents (incident reports, documentation of resulting action and feedback, etc.) provides important information on organisational learning by considering issues such as:
    • How often voluntary reporting processes are used
    • The quality and scope of incident reports, and whether important issues are covered appropriately
    • Whether reports are acted on, how feedback is communicated, and what the process for responding to reports entails
    • How trends in incident data are collected and acted upon.

This information assists the assessment team in understanding the reporting processes in place, people’s involvement in the processes and the quality of the feedback and outcomes.

These details can be collected prior to the launch of the survey or during the site visits. The subsequent interviews and workshops provide additional opportunities to collect information about the organisational structure and context, social environment and financial health.

Interviews and Workshops

The interviews and workshops are intended to collect useful and complementary information to support the ANSP Safety Culture diagnosis. During interviews and workshops, ANSP members usually serve as informants, who interact with the assessment team using their own terms and concepts to express their point of view.

It is important to run multiple workshops, since different workshop groups may not always say the same things. Current practice in EUROCONTROL recommends a total of 5-6 workshops with 4-6 participants plus external facilitators.

The workshops and interviews are designed to elicit views from ATCOs, Technical staff and management on a range of issues, including the prevailing culture through its impacts on safe behaviour, reporting of errors and incidents, and communication about safety priorities.

The enablers and motivators, and barriers and disincentives to Safety Culture emerging from these activities, provide insight into the Safety Culture dynamics of the organisation.

The final goal is to integrate the information from the workshops and interviews with the findings of the Safety Culture questionnaire, to highlight the full range of Safety Culture issues within the organisation.

Interviews are normally conducted with one person at a time. Personnel typically involved in the Safety Culture interviews include:

  • ANSP Top Management, e.g. CEO, Safety / Quality Manager, Operations Deputy Director, Technical Deputy Director, Financial Deputy Director
  • ANSP Middle Management, including ATCOs Supervisors, Maintenance Supervisors, Trainers.
  • Subcontractors (if possible)

In general, individual interviews are carried out with ANSP senior and middle managers to collect in-depth information about Safety Culture. However, additional interviews with airport and regulator management (where possible) can help to gain a wider overview of how safety is valued by all stakeholders dealing with safety within the aviation system.

Each workshop should be conducted with members of the same employee group. The ideal number of participants should not exceed 7 to 10 people. Ideally, participants will be members of different teams or shifts and of different ages, experience, etc., so that a variety of perspectives and experiences can be discussed. It can be useful to have one workshop aimed at the executive management of the ANSP (the CEO may be interviewed separately). Another workshop might comprise personnel who do not work on the front line, such as Human Resources and Security. Remaining workshops would then comprise groups of 4-6 ATCOs, engineers and shift supervisors. Each workshop requires two facilitators one of whom will take detailed notes.

Analysis and Conclusion from the Assessment

During the analysis phase, all collected raw data are analysed and interpreted, and the results collated into a report resulting from a collaborative process between the assessment team and ANSP members.

In assessing the collected data, it is important to note that the opinions and perspectives accessed during a Safety Culture assessment are generally obtained from only those personnel who interact directly or indirectly with the assessment team and may not, therefore, be representative of the views of all employees. An essential step towards presenting an accurate assessment of an organisation’s Safety Culture is therefore to compare and ‘triangulate’, or cross-check, information collected from a variety of different sources:

  • Safety Culture questionnaire / survey
  • Analysis of documents (manuals, policies, etc.)
  • Interviews with management
  • Workshops / focus groups
  • Site visits (e.g., observations, informal discussions, etc.)

The analysis is based primarily on the appraisal of the observable and analysed issues in the actual context of the organisation, that is, the ANSP’s safety outcomes and performance, and the organisational and national culture and the commercial and social environment in which the ANSP operates.

The outcomes of the analysis are used to identify and summarise strengths and weaknesses (or ‘opportunities for improvement’) in the organisation’s Safety Culture to help ANSP managers and staff to understand their Safety Culture and how they might improve it.

Typically, three to five or so major areas for improvement are identified. Within these areas there are usually some ‘quick wins’ related to communication and safety strategy, for example, or incident learning practices. Other issues may be more ‘long hand’. In all cases, it is up to the ANSP to decide which to tackle, or indeed to consider other options. The resultant ‘strategy’ needs to be ‘owned’ and driven by the company. It is essential that the ANSP has ‘ownership’ of these goals and develops improvement strategies that are realistic and practical, given the organisation’s constraints and margins for manoeuvring, as imposed by both the organisational context and national culture.

Thus, there is a need for a collaborative approach to translate the identified change needs into action plans and solutions. This can be achieved through a facilitated workshop involving the decision-makers who can assure the implementation of the agreed actions.

Safety Culture Re-assessment

Following the completion of a Safety Culture assessment and implementation of the improvement plan, it is recommended that re-assessment is conducted to:

  • evaluate the effectiveness of the improvement strategies
  • promote continuous improvement by identifying ongoing weaknesses
  • fine-tune improvement efforts to reflect changes in Safety Culture strengths and weaknesses
  • reaffirm commitment to Safety Culture improvement, and defend against other projects being given priority.

The Safety Culture assessment campaign, from the launch to the decisions for change by the ANSP management, can take about a year (see Figure 1).

Firgure 1 - Timeframe of the Safety Culture assessment and re-assessment campaigns

More time will be required before an organisation can implement the change strategies and see the early outcomes of any improvements. Therefore, a re-assessment campaign will be relevant only after two to three years.

Related Articles

Further Reading

EU

EUROCONTROL

Others

Categories
Safety Culture

SKYbrary Partners:

Safety knowledge contributed by: