The Training Assessment Plan

Assessment of students/trainees is an important part of education and training. It is usually regarded as the final phase (the end) of the training process. There are four elements which determine the training results: syllabus, training plan, training event plan and training assessment plan.

The first three of them are dealing with:

a) Training objectives classified by subjects, topics and sub-topics (syllabus);

b) Details for each subject, topic and objective, training requirements (type of training event, educational material needed, method and mode of delivery), time scale for achievement, statement of performance objectives or tests to increase the accuracy of the specifications (training plan);

c) A set of actions identified in the training plan being the smaller unit of training; association of a training technique, media, learning rate and mode of delivery to be used by the instructor when preparing and providing the training (training event plan).

The fourth element – the training assessment plan - is meant to identify, the topic and the objective of the assessment for each subject. In order to make the process more efficient, sets of assessment objectives are linked to a performance objective which is used as a basic reference for the test performance.

Competence to be Assessed in ATCO Training

In the context of ATCO training the term “Competencies” defines the necessary actions that a person is supposed to take so as to perform a job or a task well or in a satisfactory manner.

Assessing competence is about stripping jobs down into their component parts and linking together the basic elements of performance – what has to be done and to what standard (Armstrong, Barron (1998) Performance Management – The New Realities).

Assessment of ATCO Students/Trainees – a Safety Critical Issue

A competence assessment scheme is regarded as a key component for strenghtening the safety culture in an ATS operational environment. It is to promote uniformity in the application of Air Traffic Control (ATC) procedures throughout the ATS Unit and encourage the use of common approaches to solving operational problems through the establishment of “best practice” controlling techniques.

The assessment of ATCO student/trainee is intended to be a credible process with a crucial role for maintaning and setting up the best practice for compliance with safety regulations and requirments. Competence assessment of controllers must be a genuine test of a controller’s operational skill and knowledge. Although failure during competence assessment is expected to be a rare occurrence, the actual assessment must be seen by controllers as one in which they will not automatically be assessed as competent in respect of their performance.

Assessment Measures Against Training Objectives

From a general perspective the assessment has to meet the training objectives and to measure the extent to which the students/trainee performance meets the training objectives. Training objectives are stated definitions of what the student/trainee is required to achieve as a result of training. The definitions must be accurate and measurable.

In EATM training documentation developed by EUROCONTROL, several categories of objectives have been referred to (see EATM Training Progression and Concepts).

• Objective: A clear statement based on a description of the learner performance (corpus) and the limits of this performance (content). The corpus always includes an action verb to ensure that the outcome is observable and that the difficulty level is stated according to a defined taxonomy
• General objective (or goal): Describes the direction to move in, rather than a detailed quantitative objective.
• Performance objective: A clear and unambiguous statement of what a student is expected to do (behavior or Performance) with the minimum level of acceptable performance (Standard in terms of quality, quantity and time allowed for completion) and conditions under which the performance is to be carried out (Conditions) - PSC. The performance objective clearly establishes a link between the training objective and the method to assess whether this training objective has been reached.

The training content is divided into subjects, themselves divided into topics that are in turn subdivided into sub-topics. This structure is used to create and classify the objectives: one general objective is linked to each subject and one or several objectives are linked to each sub-topic. Optionally a main objective may be linked to a topic.

Levels of Competence to be Assessed (Measured)

In accordance with Benjamin Bloom, an educational psychologist, five levels of competence are identified, numbered 1 to 5, plus an initial level (named 0) for pure information. They are defined as follows:

• Level 0 ‘To be aware of’.
• Level 1 ‘To Know’ - Requires a basic knowledge of the subject. It is the ability to remember essential points; the learner is expected to memorize data and to restore it.
• Level 2 ‘To understand’ - Requires an understanding of the subject sufficient to enable the learner to discuss intelligently. The individual is able to picture for himself/herself certain objects and events, and to act upon these objects and events.
• Level 3 ‘To apply’ - Requires a thorough knowledge of the subject and the ability to apply it with accuracy. The learner should be able to make use of his/her repertoire of knowledge to develop plans and activate them.
• Level 4 ‘To analyze’ - The ability to establish a line within a unit of known applications following the correct chronology and the adequate method to resolve a problem situation. This involves the integration of known applications in a familiar situation.
• Level 5 ‘To evaluate’ - The ability to analyze new situations in order to elaborate and apply one or other relevant strategy to solve a complex problem. The defining feature is that the situation is qualitatively different to those previously met, requiring judgment and evaluation of options.

Methods of Competence Assessment

Competence may be assessed by a system of:

a) Continuous assessment: Continuous assessment may be achieved by the competence assessor observing the standard of the air traffic control service provided by those whose competence he/she will certify while working with them during normal operational duties. Where the competence assessor has not had sufficient contact with the controller to adequately assess his/her performance the assessor will not certify the controller’s competence until a dedicated practical assessment has been conducted. The controller concerned must be advised that a dedicated practical assessment is to be conducted.

b) Dedicated practical check: To conduct a dedicated practical assessment the competence assessor sits with the controller with the sole purpose of observing the quality and standard of work being carried out. This differs from continuous assessment in that the competence assessor is not involved in any tasks other than observation of the controller being assessed. The controller concerned must be briefed on the conduct of the assessment particularly when it is necessary to monitor and observe him/her from a remote position. Following a dedicated practical check the competence assessor must de-brief the controller being assessed and give feedback regarding the quality of working practices observed. If a student/trainee or controller’s performance is observed only in some working situations the assessment must be supplemented by questioning on other situations, e.g. low visibility operations, snow removal, military activity, etc.

c) Combination of (a) and (b) above; and

d) Oral examination and/or a written or Computer-Based Training (CBT) test of the controller’s knowledge of the unit specific and generic ATC procedures. The oral element of the process (when it forms part of the agreed process) is usually carried out after the practical element. All key performance objectives must be tested to confirm understanding. Scenario-type questioning allows the assessor to gather additional evidence of how a student/trainee or controller would have reacted in circumstances that were not observable but are nevertheless considered important to the overall operation at that ATS Unit. A student/trainee or controller who demonstrates practically that he can do the job (seen by direct observation) and can explain the reasons for acting in a particular way (verified through the oral element) has demonstrated understanding and has fulfilled the requirements to be considered competent. As stated, when used properly, the oral element of the process primarily establishes how the student/trainee or controller would work under circumstances the assessor was unable to witness. Additionally, the assessor will be able to focus on certain specific observations to provide clarification. The oral examination will, therefore, give a clear indication that the student/trainee or controller knows, not only what he should be doing, but why he should be doing it. The oral process requires considerable skill and it is difficult to ensure consistency between individual competence assessors. This further highlights the importance of a structured training process for competence assessors.

The Place of Written Examinations in ATCO Students/Trainee Assessment

It is easier to administer and to ensure the consistency of written than of oral examinations particularly when using multiple-choice questioning. Although multiple-choice questioning can test knowledge it is not appropriate for determining what a controller would do in a particular operational situation. The most comprehensive method of testing a controller’s understanding, as opposed to his/her possession of pure knowledge, would be a combination of multiple-choice questions, that assess the controller’s knowledge of unit specific and generic ATC procedures, conducted using computer-based training, together with a separate oral examination which tests the controller’s reactions to operational situations.

Related Articles

• Controller Training Methods and Tools
• Line Oriented Flight Training
• Technical Competencies

Further Reading

EASA

• Air Traffic Controlers' Licencisng and Certification, Consolidated version: Technical requirements and administrative procedures related to Commission Regulation (EU) 2015/340 and AMC and GM to ATCO, August 2015

EUROCONTROL

• EATM Training Progression and Concepts

FAA

• Aviation Instructor's Handbook, 2008

Others

• Bloom, Benjamin S. Taxonomy of Educational Objectives. Published by Allyn and Bacon, Boston, MA
• Armstrong, Barron (1998) Performance Management – The New Realities