Flybe Fume Event (Part 1): Compressor Wash Maintenance Human Factors Case Study (ERJ195 G-FBEJ)
On 28 February 2019 Flybe Embraer ERJ-195 (ERJ-190-200 LR / E195) G-FBEJ was commencing takeoff in Exeter, bound for Alicante, when the flight crew detected a sweet-smelling odour and observed smoke entering the cockpit. The takeoff was abandoned and after smoke and fumes were reported in the cabin an evacuation commenced. This article, the first of two, will examine the cause of the fumes. The second will examine the evacuation.
Flybe Embraer ERJ-95 (LR) G-FBEJ, Paris CDG 13 May 2019 (Credit: Olivier Cabaret CC BY 2.0
The Safety Investigation
The UK Air Accidents Investigation Branch (AAIB) explain in their safety investigation report that:
During overnight maintenance on the night before the accident, an engine compressor wash was carried out on G-FBEJ’s No 1 [GE CF34-10] engine.
A high-power engine ground run was not performed following the compressor wash, resulting in residual cleaning solution remaining in the compressor bleed air ducts. This can lead to fumes or unusual odours entering the cockpit and cabin.
There is sadly often over-enthusiasm for labelling occurrences as ‘Failure to Follow Procedures” (FFP) events and obsessing on classifying how and why personnel ‘violated’ procedures, applying crude culpability decision aids. This case study shows that approach is very misguided in limiting the potential for holistic systemic improvement.
Fortunately, the “sole objective” of AAIB safety investigations “is the prevention of accidents and incidents, without the apportionment of blame or liability”. This investigation therefore did not fall for the biases that can occur in relation to procedures.
The Safety Investigation: Compressor Wash Procedures
General Electric recommends engine cleaning to reduce contaminant build-up and maintain engine performance.
Compressor washes are performed by maintenance personnel, using a wash rig, which uses either water or a water and detergent mix.
The engine wash rig used on G-FBEJ was fitted with two pressurised fluid tanks, one which contained water and the other a water/detergent mix.
During operation, the fluid can be directed into a water-wash manifold installed on the engine, to dispense water/detergent into the compressor.
The maintenance personnel used Engine Service Manual (ESM) task 72-00-00-100-801 ‘Engine performance recovery,’ revision date 31 March 2016.
The task requires a minimum of two people, one to operate the engine and system controls in the cockpit and one to operate the compressor wash rig.
The ESM advises that:
For some environments, washing with a cleaning solution … may be more effective than washing with water only….If a cleaning solution is used, it is important to follow instructions for rinsing and drying-out the bleed systems.
A caution states:
FAILURE TO ADEQUATELY DRY THE INTERNAL ENGINE AIRFLOW COMPONENTS AFTER AN ENGINE WASH CAN RESULT IN ODOR-IN‑CABIN EVENTS WHICH HAVE CONTRIBUTED TO SITUATIONS SUCH AS AIR TURNBACKS AND ABORTED TAKEOFFS. PROPER ENGINE DRY-OUT IS IMPORTANT TO PREVENT THOSE SITUATIONS.
The ESM lists several detergents, including Turco 5884 used by the operator, which the ESM states should be mixed one part detergent to four parts water (i.e. 20% detergent). Rinsing should be with water only. The AAIB go on (our emphasis added):
One subtask describes the procedure to wash the internal engine airflow components with water only and an alternative subtask describes the procedure to do the wash with a cleaning solution. It recommends that to get the best cleaning results, two washes should be done as well as a soak period between application of the cleaner, followed by two rinses to make sure that the cleaning solution is removed.
A further subtask describes the rinse procedure and states that any remaining cleaning solution should be drained from the compressor wash rig and the fluid tanks filled with rinse solution (water).
The procedure instructs personnel to run the engine at idle for five minutes before operating the anti-ice and engine bleed systems, while the engine continues to be run at idle. The aircraft’s pneumatic and environmental control (ECS) systems use bleed air extracted from the high pressure compressor (HPC). The purpose of this procedure is to purge fluid which may be trapped in the engine bleed ducts or ECS.
However, the procedure acknowledges that running at Idle rating “cannot fully dry out the aircraft ECS”. A note says (our emphasis added):
If the ECS system can be verified to be dry or a water-wash without cleaner was done, engine run at 65 percent N1 [Low Pressure / Fan shaft speed] can be considered optional. If not, drying-out the ECS system via operation up to 65 percent N1 is recommended.
The reason this N1 is selected is that it ensures the ECS bleed switches from the High Pressure Compressor (HPC) stage 9 to stage 5, thus also drying out the stage 5 ducting.
AAIB explain that:
The procedure requires that an observer is positioned in the aircraft cabin to detect any odour or detergent after completion of the ECS and anti-ice dry-out procedure. If odours or fumes are detected, the engine dry-out should be continued until none are detected.
When consulted after the incident a number of personnel observed that on this topic the ESM was “poorly laid out and difficult to interpret” and the AAIB note it is (our emphasis added)…
…comprised of many subtasks covering several wash, rinse and drying scenarios. It contains multiple notes and cautions, not all of which stand out from the main text and some of which contain critical information.
In particular, the information relating to the need to perform a high-power engine run during the dry-out procedure is included in a note rather than in a procedural step.
As mentioned previously, the AAIB observe that on the matter of a drying run (our emphasis added):
The ESM indicated that this was a recommended rather than required action.
It is not clear if any of these issues had been recognised prior to the incident. Furthermore (our emphasis added):
The maintenance organisation typically used process sheets for complicated maintenance tasks but did not have a specific process sheet for performing compressor washes.
The Safety Investigation: Planning and Conduct of the Compressor Wash
For G-FBEJ:
The requirement for the compressor wash had been identified by the operator’s maintenance control department on the preceding afternoon, to rectify a decreasing trend in engine turbine temperature, which had been highlighted by routine engine performance monitoring.
Maintenance planning for the task did not identify the time, resources or competence required for the compressor wash task, nor confirm that the required resources were available.
The AAIB report explains:
The work had been planned in for the day shift but problems encountered with sourcing a serviceable compressor wash rig meant the task had to be carried out by the night shift. Due to the poor condition of the compressor wash rig engineers on the day shift spent considerable time getting it ready…
However, most preparations were still completed by the day shift.
This included replenishing the compressor wash rig with air, detergent and water and attaching it to the No 1 engine. Engineer 3 had undertaken these actions. He reported that prior to replenishment, the cylinder with the detergent/water mix was already approximately half full, with what he assumed to be the correct concentration of detergent/water. He then continued to add to the liquid already in the cylinder using a ratio of four parts water to one part detergent.
The investigation was not able to determine who had last used the rig and no way to conform the initial concentration was correct.
Engineer 4 accepted the aircraft into the hangar for the compressor wash. Although he attempted to check if the engineers on the oncoming [night] shift had the approval to carry out the task, the databases he consulted were out of date and he did not have the knowledge to verify the approvals by other means.
In fact, only two engineers (Engineer 1 and one other) on the night shift were qualified to conduct ground runs on the E195. Both had little actual ground run experience and were also only qualified for low power runs. Engineer 1 had more experience with the Dash 8 Q400, which only needed low power engine runs for compressor washes. He did however…
…not have any recent experience carrying out engine ground runs on the E195 and had not performed a compressor wash on an E195 before.
It was observed that:
There were only two engineers within the operator’s maintenance organisation company who were qualified to train others on conducting engine ground runs on the E195. A lack of trainers, training opportunities and access to simulators had resulted in a reduction in the number of competent authorised engineers available.
It is not clear if this had been noticed prior. The AAIB do observe though that:
Although the aircraft required a compressor wash, this was not an urgent requirement and could have been allocated to a suitably resourced shift.
The AAIB report explain that on the prior night…
Engineer 1 carried out the engine ground runs from the cockpit and certified the task. Two…personnel were positioned outside the aircraft, one to operate the compressor wash rig and communicate with Engineer 1 via a headset and the other to check the engine.
When Engineer 1 arrived at the aircraft, the compressor wash rig fluid tanks were full and it was already attached to the No 1 engine, having been attached by the day shift. One compressor wash cycle was carried out using a water/detergent mix and four rinse cycles using water.
Engineer 1 then performed the engine drying procedure using idle power only. He could not recall any warnings or cautions for the ECS when carrying out the dry-out run. Although he could smell the cleaning solution when the dry-out run began, he could no longer smell it after the run had finished but the aircraft doors had been opened to aid venting.
Engineer 1 commented that when compressor washes had previously been done by the night shift, they were part of a larger maintenance input rather than a stand-alone task; this allowed for the high-power engine dry-out runs to be carried out by the oncoming day shift.
He considered that the night shifts were generally undermanned and there was a lack of support functions that would be present during the normal working day.
It was noted that:
Engineer 2 was the hangar bay supervisor and allocated the compressor wash task to Engineer 1…without fully understanding the requirements of the task.
Information regarding approvals and authorisations held by engineers was not readily accessible by managers, shift supervisors and maintenance planners which made it difficult to plan resources and assess the capabilities of the current or oncoming shift. This information was normally accessible by staff working core office hours.
Hence, the supervisor, Engineer 2, also did not realise the night shift did not have the necessary authorised engineers to fully complete the task.
He stated that the night shift was normally pushed for time, with aircraft often not arriving in the hangar until 2200 hrs and needing to be back on-stand by 0300 hrs. He stated that he does not have the opportunity to review resources and required materials before work is carried out and it is often left to him to address any issues with the work packs, missing materials or tasks that cannot be performed.
The engineers on the night shift worked a permanent shift pattern of four 12-hour night shifts, followed by four nights off. The compressor wash was carried out on the third night of the shift and the work was performed between 2300 hrs and 0300 hrs.
The Safety Investigation: Post-Incident Maintenance and Lab Analysis
Subsequently, several additional compressor rinse cycles were conducted to flush residual detergent from the No 1 engine and HPC5 stage bleed ducts.
Despite this, detergent bubbles continued to come out of the engine during idle power engine runs, which led the operator to consider that the dilution of the cleaning solution used may have been incorrect. A sample of the detergent/water mix taken from the compressor wash rig was subjected to FTIR [Fourier Transform Infrared Spectroscopy] analysis…. This determined that the concentration of the sample was 31.3% detergent [vs the required 20%].
It could not be determined if and to what extent the compressor wash rig had been topped up during the compressor wash and rinse cycles, which could have altered the dilution from that used on G-FBEJ’s engine. Given the difficulty purging the residual detergent from the engine, and the result of the sample analysis, the operator considered that the compressor wash rig may have contained an overly strong concentration of cleaning solution, prior to being replenished by Engineer 3.
As the precise concentration of cleaning solution used could not be established, nor the effect of its long exposure to engine parts, the engine was withdrawn from service on the engine manufacturer’s recommendation.
AAIB also note that:
Many of the compressor wash rigs were found to be in poor condition. There were no records of when they were last replenished, the type or concentration of detergent used or on which aircraft they had been used.
The Safety Investigation: Flawed Change Management, Over-Reliance of Frontline Staff and Wellbeing
In the months preceding this incident, the nature and volume of the maintenance work carried out by the night shift at the operator’s base had changed considerably. Termination of a contract with an external maintenance provider had resulted in an increase in maintenance work being undertaken at the operator’s own maintenance facility.
Previously the bulk of the work undertaken was in-depth planned maintenance tasks, but in the period preceding the accident most of the maintenance undertaken was reactive with short turnaround times and little prior notice of the type of work required.
Engineers considered that the maintenance was often not correctly resourced with respect to spares, tooling or manpower and the time allocated for tasks was often incorrect and did not take account of the time needed to tow the aircraft from its location to the hangar and back.
While the operator had implemented a change management process to support the transition of maintenance from the external maintenance provider to its own maintenance facility, the internal investigation identified that procedures and processes had not been adapted to cater for the change in maintenance type.
In common with other shifts, on the night shift a small number of engineers, often in supervisory roles, held the majority of approvals and authorisations for conducting maintenance such as engine ground runs, certifying maintenance and certifying the release of aircraft to service. This placed substantial responsibility on a small number of individuals.
The operator’s own internal investigation identified that there was “a ‘can do’ culture throughout its engineering departments and a willingness to ‘get the job done'”. It’s not clear if this was known prior to the incident or only emerged from this investigation. However, in contrast to that culture:
…its change management process had endeavoured to empower engineers to call ‘stop’ if they felt they could not complete maintenance safely but in the absence of supporting procedures, these were found to place an over-reliance on the individuals.
However:
Engineer 1 and Engineer 2 disclosed that they were each experiencing personal issues which had been occupying their thoughts, but each considered that they were fit to continue working…
Although their managers had been aware of their circumstances, no specific support had been put in place, nor any restrictions on what they could supervise or certify.
Perhaps surprisingly 16 years after human factors training became a mandatory requirement in Europe:
Supervisors and managers had received only minimal training in how an individual’s welfare can affect their performance.
Additionally, no fatigue risk assessment had been conducted on night shift operations.
AAIB comment that:
Maintenance planning, both at the operator [sic i.e Continuing Airworthiness Management Organisations (CAMO)] and hangar [sic i.e Part 145 maintenance organisation] level, did not adequately identify the resources required to undertake the compressor wash, nor attempt to match the requirements of the task to the capabilities of the oncoming hangar shifts.
Systems in place did not assist maintenance planners and managers to easily establish the competence and approval status of individual engineers. As a result, this maintenance task was allocated to a shift which did not have the correct competence and approvals to carry out and certify the task, or to release the aircraft to service.
Had Engineer 2 fully understood the requirements of the task, the resources required to complete it and the approval status of the engineers on the night shift, it is likely that the task would have been rejected by the night shift instead of being allocated to Engineer 1.
Similarly, had Engineer 4 understood these aspects, it is likely the task would have been deferred when it could not be completed by the day shift.
All were working within an environment where organisational changes were occurring at a time the airline was under financial pressure (the operator went into administration and ceased operating in March 2020). It’s not clear from the safety investigation report if the financial situation was relevant.
The CAMO’s oversight of the maintenance organisation, the maintenance organisation’s quality system and occurrence reporting and the oversight by the regulator, the UK CAA, are not examined.
Safety Actions (Airworthiness)
The AAIB report that the operator’s maintenance organisation:
- Enhanced the control and tracking of maintenance ground support equipment to enable calibration expiry dates to be managed more effectively.
- Introduced a maintenance planning procedure so that maintenance requirements are identified earlier in the working day to allow appropriate resources to be identified and allocated
- Undertook a review of tasks performed within the hangar to identify specific training requirements with a view to developing training programmes.
- Launched an engineer’s competency passport scheme to enable maintenance planning departments to allocate specific maintenance tasks to maintenance stations where the correct resources are available.
- Introduced additional simulator training for engineers to undertake engine ground runs and committed to review its recency period for conducting engine ground runs.
- Introduced a programme to verify that engineers have the correct procedures, records, equipment and tooling, personnel requirements, approvals, replacement parts, environment and information before commencing a maintenance task.
- Committed to undertake fatigue risk assessments for night shift maintenance personnel and initiated an engineer welfare programme.
- Updated its change management process to ensure appropriate management of the risks associated with the changing nature of maintenance being conducted in its base hangar.
The operator ceased operations in March 2020 before these actions were complete. Additionally, (emphasis added):
In June 2020, the engine manufacturer updated ESM subtask 72-00-00-410-004 to require, rather than recommend, that a high-power engine dry-out run is conducted after a compressor wash using detergent.
AAIB Conclusions on the Maintenance Aspects
A lack of maintenance planning, training and control of resources led to an undesirable situation where a maintenance task was allocated to an engineer who was neither qualified nor competent to complete the task.
A key step in the engine drying procedure was only described as ‘recommended’ and the engineer did not complete all the elements of the task. This resulted in residual cleaning solution remaining within the ECS system, causing smoke and fumes within the cabin and cockpit and leading to an emergency evacuation. The [OEM] engine drying procedure has since been amended to require this step to be carried out.
Previous Incidents
In 2017, we wrote: Smoke in Cabin: Anatomy of a Wash Rig Error. This was based on the Irish Air Accident Investigation Unit (AAIU) report on a smoke in cabin event on Aer Lingus Airbus A320-214 EI-DVJ at Dublin on 3 October 2015. In that case the wrong fluid was added to the wash rig, but the incident clearly showed the potential for a lack of knowledge on how to complete tasks with specialist equipment. In that case we also noted:
This incident again questions the reliance on theoretical classroom HF training (see also: Aircraft Maintenance: Going for Gold? and Airworthiness Matters: Next Generation Maintenance Human Factors) and Continuation Training (see: Does Continuation Training Change Behaviours?).
It would be fascinating to know if Flybe had routinely considered the ‘free lessons’ from such external events.
We also wrote Dash 8 Q400 Return to Base After Pitot System Contaminated By Unapproved Test Kit Lubricant after another Flybe incident which occurred on 15 November 2018. In that case DHC Dash 8 Q400 G-JECR returned to Exeter, shortly after departing for Paris Charles de Gaulle when the pilots received an intermittent ALT MISMATCH message during the climb. The AAIB investigation revealed a difference between ‘work as imagined’ and ‘work as done’ in testing the pitot static system and teh use of specialist equipment. The AAIB concluded that:
Following scheduled maintenance…a small quantity of a silicone-based grease was blocking three of the four static pressure holes of the left primary pitot static probe. [This] resulted in an altitude mismatch of 140 ft between the commander’s and co-pilot’s altimeter. This may have been caused by using a non-approved grease to aid sealing the test adaptor to the pitot static probe, a task which can sometimes be problematic. The kit manufacturer’s recommended lubricant is sometimes missing from the kits and the AMM and the kits instructions do not include any details on installation or sealing.
Subsequently, the company…
…purchased new air data accessory kits and implemented tighter tool control of the kits to ensure all the components are always available.
That safety investigation report also does not consider prior regulatory oversight or maintenance quality system performance.
Safety Resources
You might find these safety / human factors resources of use:
- James Reason’s 12 Principles of Error Management
- Back to the Future: Error Management
- This 2006 review of the book Resilience Engineering by Hollnagel, Woods and Leveson, presented to the RAeS by Aerossurance’s Andy Evans: Resilience Engineering – A Review and this book review of Dekker’s The Field Guide to Understanding Human Error: The Field Guide to Understanding Human Error – A Review.
You may also find these Aerossurance articles of interest:
- B1900D Emergency Landing: Maintenance Standards & Practices
- ERJ-190 Flying Control Rigging Error
- Luftwaffe VVIP Global 5000 Written Off After Flying Control Assembly Error
- C-130 Fireball Due to Modification Error
- Fatal $16 Million Maintenance Errors
- Engine Shutdown Results in Revised SOV Rigging Instructions
- SAR AS365N3 Flying Control Disconnect: BFU Investigation
- ‘Crazy’ KC-10 Boom Loss: Informal Maintenance Shift Handovers and Skipped Tasks
- Maintenance Issues in Fire-Fighting S-61A Accident
- United Airways Suffers from ED (Error Dysfunction)
- NTSB Confirms United Airlines Maintenance Error After 12 Years
- USAF RC-135V Rivet Joint Oxygen Fire
- ATR72 VH-FVR Missed Damage: Maintenance Lessons
- The Missing Igniters: Fatigue & Management of Change Shortcomings
- A319 Double Cowling Loss and Fire – AAIB Report
- B747 Landing Gear Failure Due to Omission of Rig Pin During Maintenance
- When Down Is Up: 747 Actuator Installation Incident
- Maintenance Human Factors in Finnish F406 Landing Gear Collapse
- Hoist Assembly Errors: SAR Personnel Dropped Into Sea
- Crossed Cables: Colgan Air B1900D N240CJ Maintenance Error
- A Lufthansa MD-11F Nose Wheel Detached after Maintenance Error
- Inadequate Maintenance, An Engine Failure and Mishandling: Crash of a USAF WC-130H
- UPDATE 15 March 2021: ATR 72 Rudder Travel Limitation Unit Incident: Latent Potential for Misassembly Meets Commercial Pressure
HindSight 30 is focused on the theme of Wellbeing. The Flight Safety Foundation (FSF) has recently published An Aviation Professional’s Guide to Wellbeing.
Flight Safety Foundation BARS Maintenance Observation Programmme (MOP)
Aerossurance worked with the Flight Safety Foundation (FSF) to create a Maintenance Observation Program (MOP) requirement for their contractible BARSOHO offshore helicopter Safety Performance Requirements to help learning about routine maintenance and then to initiate safety improvements:
Aerossurance and its partners can provide proven, practical expertise to successfully implement a MOP, introduce an effective competence assessment process or analyse the hazards of maintenance tasks before an occurrence.
Recent Comments