Description

On 7 January 2013, almost immediately after all passengers and crew had disembarked from a Japan Airlines Boeing 787-8 (JA829J) which had just arrived at Boston from Tokyo Narita as JA 008, cleaning staff reported an electrical burning smell and dense smoke in the aft passenger cabin to a maintenance engineer who was in the flight deck. It was quickly found that the source of the smoke was the Lithium-Ion APU battery located in the aft electrical equipment bay where dense smoke and subsequently flames were seen coming from the APU battery case. ARFF intervention successfully controlled the fire with a minor burn injury sustained by one fire fighter.

Investigation

An Investigation was carried out by the National Transportation Safety Board (USA) (NTSB).

Although it did not materially hinder the conduct of this particular investigation, it was found that there were data recording deficiencies in respect of both the Flight Data Recorder (FDR) and Cockpit Voice Recorder (CVR) elements of both Enhanced Airborne Flight Recorders (EAFR) fitted to the aircraft. It was found that the FDR elements had ceased recording valid data during the previous flight and had instead - in accordance with the unit design - continued to repeat the last valid readings of the recorded parameters - described as 'stale data' by the Investigation. It was also found that the data recording quality on the CVR elements was poor so that although this did not affect the ability to replay an intelligible recording with the engines shut down - the relevant action in this investigation, with the aircraft in flight, much of the recording was obscured by ambient noise, with the effect only reducing with engines running on the ground.

The occurrence of an apparently related in-flight incident involving the same battery type fitted to another Boeing 787-8 nine days later was noted and the NTSB assisted in the investigation of that event which led to the publication of a Final Report by the Japan Transport Safety Board on 25 September 2014. This Report concluded that the aircraft main battery had been destroyed as a consequence of thermal runaway after a short circuit in one of the cells and considered that the certification process for the aircraft type had "underestimated" the effects of such a situation.

The Investigation also noted the failure, on 14 January 2014, of the main battery of a Japan Air Lines 787 whilst it was parked at the gate at Tokyo Narita and being prepared for a service. In this case, preliminary indications are that one battery cell had overheated and vented electrolyte which was contained, as expected, within the battery case enclosure introduced after the events in January 2013.

The FAAs initial response to the event being investigated here in the form of the announcement on 12 January that a comprehensive review of the 787 Electrical Power System (EPS) and its interaction with aircraft mechanical systems would be undertaken was noted. Further FAA action four days later, following the occurrence of the second similar event in Japan earlier the same day, to ground the US 787 fleet until corrective action to be determined had been formulated, published and complied with was also noted. Other operators of the type worldwide grounded their 787s to await developments. The required corrective action was subsequently detailed in FAA AD 2013-08-12 issued on 26 April 2013.

An Interim Factual Report detailing progress with the NTSB Investigation was issued on 7 March 2013. An Appendix to this Report providing a copy of the FAAs Type Certification Special Conditions for the Boeing 787 which had been issued in 2007 in place of the normally applicable requirements for electrical equipment and installation requirements contained in 14 CFR 25.1353. was also included in the Final Report as Appendix 'B'.

At this point, it had been established that APU use prior to and during the preceding flight had been in line with normal procedures - for about 30 to 40 minutes on the gate at Narita until after engine start and then after landing at Boston when it had been left running when the flight crew departed the aircraft 15 minutes after gate arrival and the airline station maintenance manager and a mechanic boarded along with cleaning staff.

Shortly after this, one of the cleaners had come to the maintenance manager who was in the flight deck and advised that there was “an electrical burning smell and smoke in the aft cabin.” The maintenance manager then noticed that electrical power to systems powered by the APU had been lost and that the APU had automatically shut down and so he selected both APU and main battery switches to the ‘off’ position. FDR data showed that the APU battery had failed about a minute after the flight crew had left the aircraft and that the APU had shut down 22 seconds after that.

Following an initial investigation of the aft electronic equipment bay, within which thick smoke was developing, and an unsuccessful attempt to extinguish the fire at the front of the APU battery case using a hand held dry chemical extinguisher, the ARFF were called. An extinguishant effective in controlling the spread of a lithium ion battery fire, supplied as the product ‘Halotron’, a Halon 1301 replacement based on CH₂FCF₃, CHF₂CF₃ and CO₂, was applied to the clearly overheated battery and its container and after a series of applications directed through the thick smoke with the aid of a thermal imaging camera, the fire and smoke finally ceased and the remains of the battery were subsequently removed from the aircraft.

The area which had contained the failed battery was examined and found to show damage consistent with both heat generation and the discharge of smoke, hot gases and electrolyte. Thermal damage was limited to an area no more than 50 cm from the battery installation and no aircraft primary structure was found damaged.

The Investigation noted that the failed battery was installed in the rear electrical equipment bay to facilitate starting of the APU either on the ground or in the air and that the main aircraft battery in the forward electrical equipment bay was of exactly the same type. This main battery functions as the primary source of power for selected electrical equipment under both normal and failure conditions. The location of the two batteries and an annotated view of one are shown in the diagrams below taken from the Official Report. Each 75 ampere hour battery has eight lithium-ion cells which have a nominal voltage of 3.7 - the first use of such a large cell size in this type of aircraft battery.

Comprehensive analysis of all the available evidence led to the conclusion, by elimination of all other potential causes, that the malfunctioning of the cell which initiated the thermal runaway had been caused by an internal short circuit of unknown origin.

Since the battery was being used in accordance with approved operating and airworthiness instructions, the applicable FAA Certification process, in particular the Special Conditions for the certification of both the battery and its charging system, as well as all the supporting documentation relating to the safety assessment of the Electrical Power System (EPS) design as a whole, soon became the main focus of the Investigation.

In respect of Boeing's use of external suppliers for the detailed design and manufacturing of the EPS, it was established that Boeing had responsibility for overall integration and certification of the equipment used in the electrical power conversion subsystem. Boeing had contracted Thales Avionics Electrical Systems to design this subsystem which included the main and APU batteries and Thales had then subcontracted various manufacturers to produce the necessary components. These manufacturers included GS Yuasa Corporation, which developed, designed, and manufactured the main and APU batteries.

To achieve aircraft type certification, Boeing had been required to demonstrate that the 787 design met the nine explicit requirements for Lithium Battery installation in the FAA Special Conditions 25-359-SC. To provide assurance that these requirements had been addressed, Boeing had carried out a Safety Assessment to determine the potential hazards that various failure conditions of the components in the Electrical Power System (EPS) could expose the aircraft and its occupants to.

The Investigation found that Boeing had "determined that the rate of occurrence of cell venting for the 787 battery would be about 1 in 10 million flight hours". However, it noted that at the time of the investigated and the related events, both of which involved the precipitation of thermal runaway after failure of a single cell, the in service 787 fleet had accumulated less than 52,000 flight hours.

Given the significant ingress of smoke to the passenger cabin during the investigated event, the potential impact of smoke generation within the aircraft passenger cabin and flight deck environment at any time when the aircraft might be occupied and the doors closed was considered. It was noted that the system for the removal of smoke and fumes from the vicinity of the failed battery was predicated on the presence of smoke detectors to trigger the dispersal of smoke overboard by fans in the cooling ducts and by a change in the position of the electrically-driven air supply valves. However, in the absence of electrical power, this process did not work, hence the build up of smoke which had occurred in both the affected bay and in the cabin area above it.

The nature of oversight of the supply chain from the battery supplier and the EPS lead contractor Thales which had prevailed prior to the investigated event was examined. It was found that Boeing surveillance of Thales had been conducted "in accordance with contractual specifications and requirements" with Boeing relying on an independent auditor to perform surveillance assessments of Thales twice a year. It was found that Thales had conducted two audits of battery supplier GS Yuasa between the time that battery production began and the failure events. These audits, in June 2011 and September 2012 had found 11 discrepancies, none of which had been directly related to battery or cell manufacture and all of which were reported to Boeing and had been subsequently closed. Boeing had not been conducting any audits of GS Yuasa, relying instead on Thales. However, afterwards, they sent their own audit team to both Thales and GS Yuasa to review the management of sub-tier suppliers, the quality of their manufacturing and business processes and their adherence to Boeing standards. This audit found 17 instances of non-compliance with Boeing requirements. At GS Yuasa, these involved failure to follow written procedures and communications from Thales and Boeing in respect of authorisation for proposed procedural and testing changes for the battery. At Thales, they involved adherence to the contractual requirement to obtain Boeing approval for changes to drawings or procedures.

Analysis of the accumulated evidence on the battery failure and its wider context led to the identification of five Safety Issues relating to the safe installation and use of large lithium-ion batteries as main aircraft electrical power sources. The text of the Safety Issues provided below is an exact quotation of the NTSB report:

• Cell internal short circuiting and the potential for thermal runaway of one or more battery cells, fire, explosion, and flammable electrolyte release.

The investigated incident involved an uncontrollable increase in temperature and pressure (thermal runaway) of a single APU battery cell as a result of an internal short circuit and the cascading thermal runaway of the other seven cells within the battery. This type of failure was not expected based on the testing and analysis of the main and APU battery that Boeing performed as part of the 787 certification program. However, GS Yuasa did not test the battery under the most severe conditions possible in service, and the test battery was different than the final battery design certified for installation on the airplane. Also, Boeing’s analysis of the main and APU battery did not consider the possibility that cascading thermal runaway of the battery could occur as a result of a cell internal short circuit.

• Cell manufacturing defects and oversight of cell manufacturing processes.

After the incident, the NTSB visited GS Yuasa’s production facility to observe the cell manufacturing process. During the visit, the NTSB identified several concerns, including foreign object debris (FOD) generation during cell welding operations and a post-assembly inspection process that could not reliably detect manufacturing defects, such as FOD and perturbations (wrinkles) in the cell windings, which could lead to internal short circuiting. In addition, the FAA oversight of Boeing, Boeing’s oversight of Thales, and Thales’ oversight of GS Yuasa did not ensure that the cell manufacturing process was consistent with established industry practices.

• Thermal management of large-format lithium-ion batteries.

Testing performed during the investigation showed that localized heat generated inside a 787 main and APU battery during maximum current discharging exposed a cell to high-temperature conditions. Such conditions could lead to an internal short circuit and cell thermal runaway. As a result, thermal protections incorporated in large-format lithium-ion battery designs need to account for all sources of heating in the battery during the most extreme charge and discharge current conditions. Thermal protections include

(1) recording and monitoring cell-level temperatures and voltages to ensure that exceedances resulting from localized or other sources of heating can be detected and addressed before cell damage occurs and

(2) establishing thermal safety limits for cells to ensure that self-heating does not occur at a temperature that is less than the battery’s maximum operating temperature.

• Insufficient guidance for manufacturers to use in determining and justifying key assumptions in safety assessments.

Boeing’s EPS safety assessment for the 787 main and APU battery included an underlying assumption that the effect of an internal short circuit within a cell would be limited to venting of only that cell without fire. However, the assessment did not explicitly discuss this key assumption or provide the engineering rationale and justifications to support the assumption. Also, as demonstrated by the circumstances of this incident, Boeing’s assumption was incorrect, and Boeing’s assessment did not consider the consequences if the assumption were incorrect or incorporate design mitigations to limit the safety effects that could result in such a case. Boeing indicated in certification documents that it used a version of FAA Advisory Circular (AC) 25.1309, “System Design and Analysis” (referred to as the Arsenal draft), as guidance during the 787 certification program. However, the analysis that Boeing presented in its EPS safety assessment did not appear to be consistent with the guidance in the AC. In addition, Boeing and FAA reviews of the EPS safety assessment did not reveal that the assessment had not

(1) considered the most severe effects of a cell internal short circuit and

(2) included requirements to mitigate related risks.

• Insufficient guidance for FAA certification engineers to use during the type certification process to ensure compliance with applicable requirements.

During the 787 certification process, the FAA did not recognize that cascading thermal runaway of the battery could occur as a result of a cell internal short circuit. As a result, FAA certification engineers did not require a thermal runaway test as part of the compliance demonstration (with applicable airworthiness regulations and lithium-ion battery special conditions) for certification of the main and APU battery. Guidance to FAA certification staff at the time that Boeing submitted its application for the 787 type certificate, including FAA Order 8110.4 Type Certification did not clearly indicate how individual special conditions should be traced to compliance deliverables (such as test procedures, test reports, and safety assessments) in a certification plan.

The other Safety Issue identified concerned a matter which was discovered as a consequence of the investigation process but did not prejudice the conduct of it or relate to the substance of the investigated event. This was formally stated as follows:

• Stale flight data and poor-quality audio recording of the 787 enhanced airborne flight recorder (EAFR). The incident airplane was equipped with forward and aft EAFRs, which recorded cockpit audio data and flight parametric data. The EAFRs recorded stale flight data for some parameters (that is, data that appeared to be valid and continued to be recorded after a parameter source stopped providing valid data), which delayed the NTSB’s complete understanding of the recorded data. In addition, the audio recordings from both EAFRs during the airborne portion of the flight were poor quality. The signal levels of the three radio/hot microphone channels were very low, and the recording from the cockpit area microphone channel was completely obscured by the ambient cockpit noise. These issues did not impact the NTSB’s investigation because the conversations and sounds related to the circumstances of the incident occurred after the airplane arrived at the gate and the engines were shut down, at which point the quality of the audio recordings was excellent.

The Investigation formally documented a number of Findings which included the following which were central to the identification and resolution of the Safety Issues related directly to the occurrence:

• The battery failure resulted from an internal short circuit that occurred in cell 5 or cell 6 and led to thermal runaway that propagated to adjacent cells.

• GS Yuasa’s cell manufacturing process allowed defects that could lead to internal short circuiting, including wrinkles and foreign object debris, to be introduced into the Boeing 787 main and auxiliary power unit battery.

• Boeing’s electrical power system safety assessment did not consider the most severe effects of a cell internal short circuit and include requirements to mitigate related risks, and the review of the assessment by Boeing authorized representatives and Federal Aviation Administration certification engineers did not reveal this deficiency.

• Boeing failed to incorporate design requirements in the 787 main and auxiliary power unit battery specification control drawing to mitigate the most severe effects of a cell internal short circuit, and the Federal Aviation Administration failed to uncover this design vulnerability as part of its review and approval of Boeing’s electrical power system certification plan and proposed methods of compliance.

The Probable Cause of the occurrence was determined by the NTSB as:

"An internal short circuit within a cell of the auxiliary power unit (APU) lithium-ion battery, which led to thermal runaway that cascaded to adjacent cells, resulting in the release of smoke and fire. The incident resulted from Boeing’s failure to incorporate design requirements to mitigate the most severe effects of an internal short circuit within an APU battery cell and the Federal Aviation Administration’s failure to identify this design deficiency during the type design certification process."

A total of 23 Safety Recommendations were issued during and at the conclusion of the Investigation.

Five Recommendations were released on 24 May 2014 in respect of initial findings that the certification testing methods and guidance for addressing the safety risks of internal short circuits and thermal runaway on large lithium-ion batteries were "insufficient" and that more generally, there was a need for outside technical knowledge and expertise to address "inadequate" certification processes for validating the safe introduction of new aircraft technology into aircraft designs:

• that the FAA should develop abuse tests that subject a single cell within a permanently installed, rechargeable lithium-ion battery to thermal runaway and demonstrate that the battery installation mitigates all hazardous effects of propagation to other cells and the release of electrolyte, fire, or explosive debris outside the battery case. The tests should replicate the battery installation on the aircraft and be conducted under conditions that produce the most severe outcome. [A-14-032]

• that the FAA should , after Safety Recommendation A-14-032 has been completed, require aircraft manufacturers to perform the tests and demonstrate acceptable performance as part of the certification of any new aircraft design that incorporates a permanently installed, rechargeable lithium-ion battery. [A-14-033]

• that the FAA should work with lithium-ion battery technology experts from government and test standards organizations, including US national laboratories, to develop guidance on acceptable methods to induce thermal runaway that most reliably simulate cell internal short-circuiting hazards at the cell, battery, and aircraft levels. [A-14-034]

• that the FAA should review the methods of compliance used to certify permanently installed, rechargeable lithium-ion batteries on in-service aircraft and require additional testing, if needed, to ensure that the battery design and installation adequately protects against all adverse effects of a cell thermal runaway. [A-14-035]

• that the FAA should develop a policy to establish, when practicable, a panel of independent technical experts to advise on methods of compliance and best practices for certifying the safety of new technology to be used on new or existing aircraft. The panel should be established as early as possible in the certification program to ensure that the most current research and information related to the technology could be incorporated during the program. [A-14-036]

The individual context for all these recommendations can be found in the covering letter communicating them to the FAA dated 22 May 2014.

A further 18 Recommendations were made following the conclusion of the Investigation:

• that the FAA should develop or revise processes to establish more effective oversight of production approval holders and their suppliers (including sub-tier suppliers) to ensure that they adhere to established manufacturing industry standards. [A-14-113]

• that the FAA should work with aviation industry experts to develop or modify design safety standards for large-format lithium-ion batteries to require that sources of excessive heating, including electrical contact resistance from components and connections, be identified, minimized, and documented as part of the design. The standards should include measures for identifying and minimizing potential sources of heating that consider the range of operating temperatures and the most extreme electrical currents that the battery could be expected to experience during repeated charge and discharge cycles. [A-14-114]

• that the FAA should work with aviation industry experts to develop or modify existing safety standards related to the design of permanently installed lithium-ion batteries to require monitoring of individual cell temperature and voltage and recording of exceedances to prevent internal cell damage during operations under the most extreme operating temperatures and currents. [A-14-115]

• that the FAA should, once the guidance requested in Safety Recommendation A-14-115 has been issued, require type certification applicants to demonstrate that the battery monitoring system maintains each individual cell within safe temperature limits at the most extreme battery operating temperatures and the heaviest electrical current loads approved for operation. [A-14-116]

• that the FAA should work with lithium-ion industry experts to

(1) conduct research into battery monitoring system technologies that could improve the recognition of conditions leading to thermal runaway,

(2) develop active mitigation of such conditions to minimize damage, and

(3) update design and safety standards accordingly. [A-14-117]

• that the FAA should Work with industry experts to develop appropriate test methods for determining the initial point of self-heating in a lithium-ion cell to establish objective margins of thermal safety for future battery designs. [A-14-118]

• that the FAA should provide their certification engineers with written guidance and training to ensure that (1) assumptions, data sources, and analytical techniques are fully identified and justified in applicants’ safety assessments for designs incorporating new technology and (2) an appropriate level of conservatism is included in the analysis or design, consistent with the intent of Advisory Circular 25.1309 (Arsenal draft). [A-14-119]

• that the FAA should, during annual recurrent training for engineering designees, discuss the need for applicants to identify, validate, and justify key assumptions and supporting engineering rationale used in safety assessments addressing new technology. [A-14-120]

• that the FAA should develop written guidance for their certification engineers and engineering designees about the use of traceability principles to verify that the methods of compliance proposed by type certification applicants for special conditions involving new technology are correct and complete. [A-14-121]

• that the FAA should, once the guidance requested in Safety Recommendation A-14-121 has been issued, provide training to their certification engineers and engineering designees on the subjects discussed in the guidance. [A-14-122]

• that the FAA should require applicants to discuss key assumptions related to safety-significant failure conditions, their validation, and their traceability to requirements and proposed methods of compliance during certification planning meetings for type designs involving special conditions. [A-14-123]

• that the FAA should require Boeing 787 operators to incorporate guidance about the enhanced airborne flight recorder stale data issue in their maintenance manuals to prevent stale data from being used for maintenance activities or flight recorder maintenance. [A-14-124]

• that the FAA should evaluate whether the recording of stale data by the Boeing 787 enhanced airborne flight recorder, including whether the data are specifically identified as stale, impacts the certification of the recording system regarding the ranges, accuracies, and sampling intervals specified in 14 Code of Federal Regulations Part 121 Appendix M, and take appropriate measures to correct any problems found. [A-14-125]

• that the FAA should Require Boeing to improve the quality of

(1) the enhanced airborne flight recorder radio/hot microphone channels by using the maximum available dynamic range of the individual channels and

(2) the cockpit area microphone airborne recordings by increasing the crew conversation signals over the ambient background noise. [A-14-126]

• that the FAA should either remove the current exception to European Organization for Civil Aviation Equipment ED-112A, “Minimum Operational Performance Specification for Crash Protected Airborne Recording Systems”, chapter I-6 in Technical Standard Order 123B, “Cockpit Voice Recorder Equipment”, or provide installers and certifiers with specific guidance to determine whether a cockpit voice recorder installation would be acceptable. [A-14-127]

• that the Boeing Company should develop or revise processes to establish more effective oversight of their suppliers (including sub-tier suppliers) to ensure that the product being manufactured adheres to established industry standards. [A-14-128]

• that the Boeing Company should modify their process for developing safety assessments for designs incorporating new technology to ensure that the conclusions made are validated and that any identified deficiencies are corrected. [A-14-129]

• that GS Yuasa Corporation should review their cell manufacturing processes to minimize or prevent defects that could affect cell safety, and ensure that your employees are properly trained to identify and eliminate these defects. [A-14-130]

The Final Report of the Investigation was published on 1 December 2014.

Related Articles

• Lithium-Ion Aircraft Batteries as a Smoke/Fire Risk
• Aircraft Batteries
• Fire Smoke and Fumes
• Certification of Aircraft, Design and Production
• Flight Data Recorder (FDR)

Further Reading

• NTSB Safety Recommendation Letter to the FAA , 22 May 2014
• NTSB: Primer on lithium ion battery technology
• FAA Airworthiness Directive: AD 2013-08-12, (Docket No. FAA-2013-0333; Directorate Identifier 2013-NM-080-AD; Amendment 39-17436; AD 2013-08-12)
• FAA Advisory Circular: AC No: 25-19A Certification Maintenance Requirements