Summary

Select appropriate conditions for evaluation: Evaluate day-to-day operations as well as critical situations.

Principle 6: Conditions for Evaluating Prototypes Icon: EUROCONTROL © (All rights reserved)

Usually, prototypes are tested under “lab” non-operational conditions to prove the concept and the potential rise in capacity. Equally important, however, is to demonstrate robust system performance under degraded conditions.

Evaluate day-to-day operations as well as critical situations

While Principle 5 suggested the usage of prototypes in general, this section discusses the conditions under which these prototypes should be tested.

Usually, prototypes are tested under “lab” non-operational conditions to validate new functions but also to demonstrate a rise in capacity to management and the project sponsor. This is absolutely reasonable, as every system design is an investment for the company. Therefore, the project should demonstrate that anticipated and actual benefits match. This is often done with a normal traffic volume where standard procedures apply, when all systems are working properly and when all positions are staffed. Under such conditions, it is relatively easy to demonstrate almost anything.

Actually, the real world is far from idealised. The main question for HF/E is: Is the system still able to put those anticipated benefits into effect, once confronted with the ruthless complexity that we experience in day-today operations? Therefore, prototypes should also be tested under abnormal conditions like extreme (high or low) workload, emergency situations, system failures and short-staffed situations.

However, a test under degraded mode or abnormal operations should not be conducted under a safety case perspective, i.e. to identify what might go wrong. It is far more interesting to see how controllers adapt in these situations and how they are able to manage the growing complexity with the help of the system. What mechanism and strategies do controllers apply? Which redundancies are most important? Which workarounds become crucial? Does the system support or impede these workarounds? The system should provide enough resilience to gracefully extend the performance and safety boundary. Vice versa, the system should not act as if still in normal operation by enforcing workflows or delivering information that are not valid in exceptional situations.

Even under normal conditions, a design only works well if it allows the operators to continually adjust what they are doing to fit the situation (work-as-done). Unfortunately, the workflows implemented in the system often have a tendency to emphasises work as it should be done (work-as-imagined). Problems arise if there is a mismatch between work-as-done and work-as-imagined (cf. Hollnagel, 2014). An evaluation under varying conditions can help to make these human adaptations visible and identify possible gaps between work-as-imagined and work-as-done. These findings are very valuable for system design because they enable the system design to support the operators in situations where they need it the most: in high workload situations, with lots of uncertainty, exceptional situations and degraded system support.

Characteristics of Normal and Abnormal Operation Modes

Characteristics of Normal and Abnormal Operation Modes Icon: EUROCONTROL © (All rights reserved)

Source: White Paper on Human Factors Integration in ATM System Design, EUROCONTROL, 2019

The White Paper is available on Bookshelf here: White Paper on Human Factors Integration in ATM System Design