1 Background

This briefing note addresses the non-technical skills among pilots. Its objective is to help training departments develop or optimize an assessment tool for these non-technical skills. The information in this briefing note is compatible with threat and error management (TEM) training and provides a method for structured assessment and feedback for non-technical skills in a training environment. The information can help a flight operation be more objective in the assessment of non-technical skills.

The term “non-technical skill” differentiates the associated skills from their technical counterparts such as traditional “stick-and-rudder” skills. The generic term non-technical skill comprises all crew resource management (CRM) skills and may be expanded to include new terms developed in the future that relate to such skills.

2 Introduction

Non-technical skills are defined as pilots’ attitudes and behaviors not directly related to aircraft control, systems control and standard operating procedures (SOPs). Classic examples of non-technical skills are in-cockpit authority, crew coordination and cooperation, communication, decision making, conflict and error management, stress and workload management, attention, vigilance, and confidence. In short, non-technical skills cover both the social and cognitive side of the pilot.

Technical skills and non-technical skills have significant interconnections and can strongly influence each other. Therefore, it is important that any discussion of one also consider the other. It would be unproductive to assess them separately.

Just as it is necessary to provide feedback to a trainee on technical skills performance, it is also necessary to provide feedback for non-technical skills performance. Therefore, it is essential to have a benchmark for comparisons when assessments of non-technical skills are made. It is not possible to give a crew beneficial feedback on their performance and skills without reference to a yardstick that allows them to understand the relative merit of their performance against an easily understood standard. Giving feedback to a pilot or crew on both technical and non-technical performance at the same time is important and demonstrates the interrelation between the two skill sets. This is often referred to as “total feedback.” Such feedback is more complete and better shows the pilot or crew what and why things happened within a particular operational context.

In order to benefit from “total feedback,” a methodology is needed for a structured assessment of non-technical skills. This briefing note outlines such a methodology and will guide you in some important principles needed when setting up a training system. It will also help avoid difficulties by clarifying some common misunderstandings about the assessment of non-technical skills.

3 Data

The methodology described below is based on principles that have been researched and validated. The primary source of the methodology is the Non-Technical Skills (NOTECHS) research, which involved a review of principles for many systems currently in use, including the U.S. National Aeronautics and Space Administration/University of Texas (NASA/UT) behavioral marker system. The NOTECHS research was later validated by the Joint Aviation Requirements Translation and Elaboration of Legislation (JAR-TEL) program. Many airlines use the principles outlined in these research efforts when providing feedback to pilots during CRM training.

The methodology and its principles

The methodology for assessing non-technical skills involves the application of a set of specific principles as part of crew training. There are three important points to note when using this methodology:

Some preconditions must be met before the methodology can be used successfully.

It is important to follow the principles in order to enhance objectivity, to achieve coordinated application and to prevent abuse.

“Assessment” is preferred instead of “appraisal” or “evaluation,” even though the words share a similar meaning — to measure performance against a set of standards.

Principle 1. Coupling of technical and non-technical skills

Technical and non-technical skills are strongly related, and greatly influence each other. The technical and non-technical sides of flight operations are like two sides of a coin, and it would be artificial to consider and assess them separately. Therefore, the first rule of this principle is that technical and non-technical skills must be considered together.

Another basic rule under this principle is that non-technical skills must be assessed in a flight-operational context that permits the integration of the non-technical and technical skills assessment in a realistic setting. Therefore, by definition, the “total assessment” of a pilot’s performance combines technical and non-technical skills in the flight-operational context.

Principle 2. Measurement through the technical outcome and its consequences

Using the outcome of technical exercises as the starting point for assessing non-technical skills has been shown to be an effective approach for instructors and examiners. Under this approach, the assessment of non-technical skills initially involves the measurement of the technical outcome of an exercise and its related consequences. Since poorly exercised non-technical skills can lead to errors in technical areas, examining the origins of flaws and failures in technical outcomes often highlights deficiencies in non-technical skills.

Principle 3. Observable facts and behavior as basis

Non-technical skills assessments should be based on observable facts and behaviors. Assessment of character traits should not be made since such ratings are subjective and can easily differ greatly among examiners. Assessment of character traits is also largely unproductive since it is difficult for a person to change such traits but fairly easy to change his or her behavior.

Principle 4. Define semantics clearly

Non-technical skills are often described by specific words and terms that create a non-technical skills jargon. The collection of the words and terms used, together with the description of behavior components and behavior categories, can be regarded as a “non-technical skill language.” In order to understand what is being said, one has to understand the meaning of the words or terms used. Hence, the non-technical skills jargon must be well-defined and familiar to its users.

There have been many special sets of jargon or “languages” developed over the years within specific companies. There are two main advantages for a company to develop its own vocabulary:

The company can employ terms it prefers to use and that are easily understood by its personnel.

The process of defining the semantics within the company will involve key personnel and give them “ownership” of the special language. Prescribed word usage is more likely to be implemented when those who must use it had a role in its development.

The fact that the language is explicitly and clearly defined is more important than the specific terms and exact wording included in it. Nevertheless, there are minimum requirements for any non-technical skills language, including:

• The language should cover the definitions of all relevant behaviors associated with the cognitive and social sides of the human operator

• The language should be clear and unambiguous. No matter who uses the language within the organization, the interpretations and conclusions should be the same

• Behavior components or elements should be defined and grouped into categories

• Components or elements are best described in a neutral manner without either positive or negative performance connotations

• A scale or yardstick is usually incorporated in the language definition to permit the neutral description to be expressed with a positive or negative nuance

• The language should not use container terms, which are very broad terms and are difficult to work with because they are too general to describe actions and decisions. A good language uses specific terms and wording

• The language should be readily available and fully familiar to all parties involved, such as the student, examiner, instructor, safety pilot and inspector.

Principle 5. Repetitive behavior observation required

The successful assessment of a pilot’s non-technical skills should not focus on one or a limited number of occurrences of any behavior. Observations of multiple instances of a behavior of interest are needed to make a valid assessment. It is possible to identify more enduring strengths and weaknesses by examining behaviors repeatedly to determine a stable pattern of performance.

Principle 6. Access to training of non-technical skills

When an assessment reveals a weakness in non-technical skills, remedial training should be available to the pilot to increase skill competency to the minimum standard.

4 Explanatory Notes

1. More often than not, non-technical skills have an impact on the technical outcome of an exercise. Thus, technical and non-technical skills should be assessed in relation to one another.

Example: Because of flawed anticipation, a briefing was delayed and airspeed was reduced too late, resulting in overshooting the localizer and subsequently exceeding instrument landing system (ILS) limits. Since the operational context indicates no external time pressure or other technical reason for the delays in the briefing and speed reduction, the root cause of the problem lies in non-technical skills. In this example, inadequate “anticipation” (non-technical) caused the delayed briefing which, in turn, led to the poor ILS capture (technical). Thus, given the flight operational context, it can be concluded that flawed non-technical skills caused the unsatisfactory technical outcome.

A counter-example would involve a student who anticipated well, made a timely briefing and reduced speed in advance but still exceeded the ILS limits. Contrary to the first example, this student would be judged as possessing good non-technical skills but lacking technical skills.

Principle 1 asserts that non-technical skills should never be assessed in isolation. Debriefing a student about his or her anticipation without relation to the technical outcomes of the exercise in the flight-operational context is largely meaningless. Adhering to Principle 1 is, in itself, a strong safeguard against making the mistake of assessing non-technical skills in isolation. Ultimately, including consideration of the technical outcome is the best way to measure both technical and non-technical skills.

2. Crew performance determines the outcome of a flight exercise. As with the evaluation of technical skills where individual pilot technical performance is examined, the same is done when assessing non-technical skills. An individual’s non-technical performance must be examined within the context of the total crew.

Even when an exercise shows sufficient overall crew performance, there can still be a marked difference in non-technical performance among crewmembers. One crewmember may exhibit flawed non-technical performance, but the excellent performance of the other crewmember can compensate and produce an overall satisfactory technical exercise outcome. Thus, although crew performance might be sufficient, any post-exercise debriefing should take into account any substandard non-technical skills among the involved crewmembers.

Another element that must be considered is that the good or bad individual performance of one crewmember can influence the performance of the other crewmember. Measuring this effect is extremely difficult, if not impossible. In a case where a crewmember’s functioning is threatened by the bad performance of another crewmember, it may be productive to examine how well the crewmember copes with the threat (TEM defines a threat as coming from outside the person).

When addressing how to observe and assess non-technical skills, there are some specific restrictions that must be met. Perhaps the most important is that the observer/assessor should not be part of the active crew. That is because an observer who is part of the active crew cannot objectively observe and assess non-technical skills, especially with respect to crew cooperation.

Using technical performance and the outcome of an exercise or flight phase as a means to assess non-technical skills has advantages for instructors. The assessment of the technical outcome, however, takes place after the exercise or flight phase is completed. It is therefore best if the assessment of non-technical skills is also conducted at the end of an exercise or flight in conjunction with the assessment of the technical outcome. This avoids forcing instructors to change their routine for assessing technical skills. On the other hand, the observation of non-technical skills must be continuous, just as observations of technical skills are.

3. Assessment of character traits (personality) is usually subjective and can easily differ among instructors. Therefore, it is important that character traits not be used in the assessment of non-technical skills. Another reason to avoid assessing character traits is because it is nearly impossible to change someone’s personality, even if the candidate is willing to do so after a debriefing.

Similarly, assessments of emotional attitudes are subjective and can easily differ among instructors. Assessment of emotional attitudes is open to personal interpretation of the attitude, which can lead to discrepancies between assessments. As such, emotional attitudes should not be assessed as part of non-technical skills.

Observable behaviors can be assessed objectively, especially when the behaviors to be assessed are clearly specified and well-defined. Because of their nature, observable behaviors can be discussed and debriefed in a clear and concise manner. Of the most importance is the fact that a person can change his or her behavior in either the short or long term after a discussion about performance with respect to both technical and non-technical skills. For example, in an extreme case in which the personalities of the two crewmembers clash, pilots can still exhibit acceptable behaviors for the duration of a flight, thereby maintaining safety. This type of self-control and focus on the primary task is an important part of professionalism.

Thus, only observable behavior should be assessed. This includes what pilots do, say, write and any actions that an instructor can hear and see. It is as far as instructors can go during an assessment. Instructors cannot read minds, nor can most trained psychologists. Therefore, the general rule must be: If it cannot be observed, it should not be used when making an assessment.

4. The company-standard non-technical skills language should be used in training during briefings and debriefings. This means that personnel should use well-defined terms when talking about and briefing non-technical skills. In this way, they will be able to understand each other. The same non-technical skills language terms and words should be used in reports as well.

It is possible to make changes to the language over time. Terms can be added or deleted as the need arises. When changing, adding or deleting a term, it is important to take care that the requirements stated in Principle 4 are followed.

5. Most human behavior is part of an individual’s general style. The goal of non-technical skills assessments is not to focus on isolated behaviors. Rather, the goal is to identify more permanent strengths or weaknesses that an individual may have. This is why the same type of situation or behavior must be reviewed multiple times to establish an individual’s pattern and to identify his or her strengths and weaknesses.

A general guideline for the identification of weaknesses is to look for behavior patterns that led to multiple situations in which flight safety was compromised. It is important to note, however, that there must be a clear relationship between the behavior patterns and a technical outcome.

6. The company should establish guidelines to define the point at which remedial training in certain non-technical skills is necessary. A wide variety of training techniques are available. Sometimes it will be sufficient to emphasize items during a debriefing. At other times, complete retraining will be required. Some examples of possibilities in between these extremes include watching videos, classroom training, counseling by an instructor or an additional simulator session with a different instructor.

Preconditions

Certain preconditions must be fulfilled before introducing non-technical skills assessments as part of training:

Instructors should be trained in assessing non-technical skills using proper techniques and methodology.

Calibration sessions should be planned for instructors. The first calibration session can be part of the initial training mentioned above. Calibration sessions typically consist of assessment exercises using paper cases or watching video scenarios. Instructors observe and assess these scenarios and discuss their ratings. The calibration process involves reaching a consensus about the assessment following each scenario. The chief instructor should be present to mediate based on his or her judgment in case of differences of opinion. The aim is for all instructors to arrive at the same assessments for the scenarios presented. Being trained to a uniform criterion greatly helps instructors in their future observations and assessments and better guarantees consistent assessments and feedback across instructors. At least one refresher calibration session is needed annually as calibration must be a continuous process.

A best practice is for the trainees to take an initial multi-crew course (MCC) or an initial CRM course before they become subject to non-technical skills assessments. They will then better understand the underlying concepts.

Each party involved — trainees, instructors, safety pilots and inspectors — should be familiar with the terms and words of the non-technical language.

Proper introduction to concepts should be planned, preferably using the simulator during a regular recurrent training session.

Guidelines for remedial training in non-technical skills should be in place before any assessments begin.

The consent of the pilots’ unions should be sought before introducing assessment and training techniques. The pilots are the ones who will be working with the system on a daily basis and should be involved during the set-up of any programs.

5 Key Points

• Non-technical skills are defined as pilots’ attitudes and behaviors not directly related to aircraft control, systems control and SOPs.

• It is beneficial to provide feedback for non-technical skills performance.

• Technical and non-technical skills are interconnected and should be assessed in parallel through examination of observable outcomes.

• A non-technical language should be developed within a company, and personnel at all levels should be able to understand the language and use it to communicate relevant information.

• Instructors should calibrate their assessments so that they arrive at similar conclusions when observing the same behaviors.

• NOTECHS can serve as a guide for the development of a non-technical skills language and the assessment of non-technical skills.