ERJ-190 Flying Control Rigging Error (UPDATED 25 June 2020 with Final Report)

The Portuguese accident investigation agency, GPIAAF, issued a safety investigation update on 31 May 2019 on a serious in-flight loss of control event involving Aruba registered Air Astana Embraer ERJ-190 P4-KCJ that occurred on 11 November 2018, immediately after a C2-Check (1C + 2C + 6 year) at the OGMA facility at the Alverca Air Base, in Poartugal.  This was a ferry flight back to Kazakhstan with 6 persons on board, not a maintenance check flight.

In-Flight Upset to, Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

In-Flight Upset to Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

In-Flight Upset to, Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

In-Flight Upset to Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

The aircraft was landed safely but with structural damage sufficient to make this an accident in accordance with ICAO Annex 13.  The crew had however encountered considerable difficulty controlling the aircraft, so much so they had debated ditching offshore.  However, they regained a degree of control.  After two unstabilised approaches, the crew landed runway 19L. The intended runway was 19R, but due to lateral drift they were forced to land on 19L.  One passenger fell and suffered a minor injury.

Flight History - Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

Flight History – Air Astana Embraer EJ-190 P4-KCJ (Credit: Embraer via GPIAFF)

https://youtu.be/6n4mQvO4-tE

Safety Investigation

GPIAAF confirmed that an incorrect aileron control cable system installation had occurred in both wings during the maintenance check conducted in Portugal.

Misrigged Flying Control Cable, Air Astana Embraer EJ-190 P4-KCJ (Credit: GPIAFF)

Misrigged Flying Control Cable, Air Astana Embraer EJ-190 P4-KCJ (Credit: GPIAFF)

GPIAFF note that:

By introducing the modification iaw Service Bulletin 190-57-0038 during the maintenance activities, there was no longer the cable routing and separation around rib 21, making it harder to understand the maintenance instructions, with recognized opportunities for improvement in the maintenance actions interpretation.

They comment on a problem with fault-finding that:

The message “FLT CTRL NO DISPATCH” was generated during the maintenance activities, which in turn originated additional troubleshooting activities by the maintenance service provider, supported by the aircraft manufacturer. These activities, which lasted for 11 days, did not identify the ailerons’ cables reversal, nor was this correlated to the “FLT CTRL NO DISPATCH” message.

Its possible, but not confirmed or discussed by GPIAAF, that the 11 days of fruitless fault-finding may have resulted in a delay and pressure to release the aircraft. GPIAFF comment that “deviations to the internal procedures” occurred within the third-party Part-145 maintenance organisation that “led to the error not being detected in the various safety barriers designed” in the process. They do not comment on what deviations occur, why they occurred or crucially how clear and practical those internal procedures were.  They do say that the maintenance organisation is making changes.  The investigators also suggest that the Type Certificate Holder review their documentation too. They also note that the error “was not identified in the aircraft operational checks (flight controls check) by the operator’s crew.” They add that:

Amongst other issues, the safety investigation will look into the aircraft design and functioning, the crew procedures and human factors aspects raised, and will now focus on the maintenance procedures and applied aircraft technical documentation, as well as the human and organizational factors involved in the Part-145 service provider.

Final Report (UPDATE 25 June 2020)

The GPIAA final report was issued 24 June 2020.

The OEM performed assessments on the aircraft primary structure showed exceedance of the limit loads, and according to the structural evaluation that was conducted based on 14CFR and RBAC Part 25 airworthiness requirements §25.305 (a), some degree of detrimental permanent deformation was found on the aircraft.

p4kcj ej190 cable fdr

The aircraft was considered beyond economic repair.

p4kcj ej190 cable damage

The OGMA quality system was found to be “well structured and detailed”, however…

Relevant gaps were…identified, namely on effective implementation and knowledge of these same procedures and standards by the different personnel in the quality system, from the least qualified operational up to the top managers, the investigation found significant gaps, namely at the management and supervision processes…

Although the technicians directly involved in the event were duly certified and authorized, the investigation identified several gaps within the team members in the training, technical and procedural knowledge, on roles as planning, scheduling, work distribution and control with direct contribution to the maintenance event.

…it was not clear how the organisation identified the organisational or soft skills training needs to the operational staff and management, to face the recognized skills and experience deficiencies in each department assumed by the middle management.

Competence assessment is mandatory in EASA Part 145.  145.A.30 (e), AMC 145.A.30 (e) e GM2 145.A.30 (e), establish amongst others that planners, mechanics, specialized services staff, supervisors and certifying staff are assessed for competence by ‘on the job’ evaluation and/or by examination relevant to their particular job role within the organisation before unsupervised work is permitted.

GPIAAF comment (emphasis added) that:

Effective competence management should ensure that each and every employee has all the prerequisites to deliver effectively the work which they need to do.  Training provision for all staff that have a production exposure is a starting point only. When implementing real aircraft maintenance competence assessment & training programs the approach needs to cover the person, the team, the organization, environmental and the required job in a defined time frame.

The AMC1 145.A.30 (e) Personnel requirements from (EU) No 1321/2014 states that competence should be defined as a measurable skill or standard of performance, knowledge and understanding, taking into consideration attitude and behaviour.

The evidence gathered by the investigation regarding AMO competency assessments suggests that these assessments were carried out from a perspective of legal compliance. This means that there is a system in place, but it is only built as an administrative procedure supported on legal minimum requirements, far from the opportunity of delivering best practice which ensures effective aircraft maintenance competence assessment and training.

Furthermore:

Interfaces between production support areas (material provisioning, task planning and sequencing and engineering) were unclear regarding responsibilities.

The human and material resources allocated to the production line was not structured, requiring constant adjustments to the planning activities, with significant interruptions in the production flow and the consequent staff/resources reallocation or complete team allocation, as verified during modification implementation tasks (SB) on the event aircraft or the aforementioned troubleshooting process.

The AMO had defined the concept of independent inspection in its internal normative [procedures]…

GPIAAF however, found that independent inspections had not been conducted in accordance with these procedures and that…

…none of the technicians interviewed by the investigation knew the scope and philosophy of this error detection procedure.

We note one problem appeared to be confusion with another term in use ‘Dual Checks’. GPIAAF highlight that:

The events sequence identified by investigation reveals, among other problems, the lack of effective safety barriers implementation in the production process, all of them perfectly defined in the regulations and on the specific maintenance procedures. The implemented system with the demonstrated aeronautical culture weaknesses allowed a maintenance action, not identified as a deviation by the error capture system, to be hidden covered with a supposed technical failure of a component or equipment.

p4kcj ej190 cable mx history

The referred aircraft design configuration deviation (cable inversion) was not detected in the numerous modifications, tasks and sub-tasks performed on the aircraft system by different technicians with double-check (DC), including the dedicated team, who worked in detail on the affected system for 11 days, in an attempt to solve the aircraft non-dispatch message.

It is then probable, that the aircraft did not present any issues that justifies the FLT CTR NO DISPATCH condition; it was simply not possible to eliminate this message due to the CBIT not completed validation routine and clear the FCM NVM during the RTS procedure.

GPIAAF also commented on weaknesses in the maintenance data for the aircraft type.

…figures, without graphical differentiation usage for proper highlight cable routing, were considered to be confusing by the involved technicians and other experienced staff on the aircraft type…

In the maintenance manual, figures were identified that intend to represent the same components with inverse representations, as is the case with the cable adjustment task compared to the installation task, as shown on the following figure. It should be noted that the referenced figures contained in the cables installation task contained correct components representation, however with presenting format limitations.

p4kcj ej190 cable mm example

After the event and during the investigation, the OEM made important changes to the approved instructions and procedures, such as changing aileron control cables replacement task adding clearer instructions and using figure colouring on SB 190-57-0038 aileron control cables – fairleads and grommet replacement related instruction, the SB for cable material replacement and also on the operational task to check the ailerons position.

p4kcj ej190 cable mm after

The SB 190-57-0038 was revised to be harmonized with AMM and the revision 3 is already available with procedures where is not required to remove both aileron control cables at the same time.

The manufacturer developed, in addition to the visual graphic representation also improved in the maintenance manual, a new CMC page as an aid to the operational test of the ailerons by AMM 27-10-00-710-801-A, which is included in the new software to be loaded onto the aircraft and which assures a validation of the effective condition of the relative positions between the command given in the control yoke and the actual position of the aileron surfaces.  This new feature already available to be included to the CMC in addition to the visual check and Synoptic check, if installed, aids the mechanic to do the Aileron Control System Operation Test.

OGMA to its credit had started in 2013 to introduce a Safety Management System (SMS) ahead of a regulatory requirement to do so.

The investigation had access to evidence that some work has been done on reporting culture best practices dissemination…

However:

From the data to which the investigation had access, the number of voluntary reports and, mainly, those of safety concern occurrences are few and incipient in the scope.

From interviews conducted of personnel at different organization hierarchical levels, flaws were found on the safety management (SMS) philosophy understanding and implementation, including at leaders’ level. During data gathering by the investigation, the just culture* practices were not demonstrated to be applied in the company.

* An atmosphere of trust in which people are encouraged for providing essential safety-related information, but in which they are also clear about where the line must be drawn between acceptable and unacceptable behavior. (James Reason 1997).

The work documents records discrepancies identified by the investigation were mainly regarding the performed sequence (inconsistent dates), misunderstanding between double-check (DC) and independent inspections, revealing a lack of understanding of the technicians and their quality system in the independent inspection concept and, not least, the regular closing procedure or so-called tasks maintenance release by the team leaders (TL).

The trivialization of working documents signing at different times of their completing and/or, eventually, performed and signed-off by different technicians is a demonstrative example, among others found by the investigation, of the organizational culture environment and respective activities risk control.

The GPIAAF also comment that:

The operator had foreseen the control of critical tasks in its engineering procedures manual (EPM-KC/EPM from 01 JUN2018). There is no evidence that the operator has verified the requirement on the Part 145 [organisation] or that the operator’s certification authority has verified or audited it.

GPIAAF Probable Cause

The investigation determined as most probable cause for the accident the improper aileron cable installation on both semi-wings during maintenance activities and subsequent inadequate independent inspection to the
aircraft flight control systems, which resulted in a reversal operation of the aircraft aileron system and led to loss of control in-flight.

GPIAAF Contributory Factors

  1. Insufficient functioning of the AMO quality assurance and failure to implement an effective production oversight, including but not limited to the independent inspection procedures to detect maintenance escapes;
  2. The AMO lack of proper risk management, based on an effective SMS program with maintenance escapes reporting system;
  3. The lack of maintenance teams properly managed and organized by skills and specialities, with the necessary training and experience;
  4. Weaknesses in the aircraft design, referring to the ailerons control system configuration, allowing the inverted cables installation in both PCU quadrants;
  5. The aircraft manufacturer’s inaccurate depiction, complex and limitations on presentation of the aileron cables routing maintenance publications;
  6. The aircraft manufacturer’s lack of proper maintenance instructions to detect the aircraft misconfiguration during the flight controls return to service procedure;
  7. The lack of proper guidance or best practices from the OEM to the operator and AMO regarding aircraft acceptance and return to service, after a heavy or complex maintenance event;
  8. The poor supervision of the maintenance organisation by the operator;
  9. Deficient maintenance organisation oversight by the supervising authorities, namely on the internal maintenance supervision procedures and occurrence reporting;
  10. The lack of proper guidance from the operator to the crew regarding aircraft acceptance after a heavy maintenance event;
  11. The crew failure to detect the ailerons misconfiguration during the operational check procedures following limited SOP’s on the highly routine pre-flight checks.

UPDATE 29 June 2020: Flight International have choosen to focus on OGMA’s reponse to the draft report (documented in the final report): Maintenance firm clashes with inquiry over severe E190 loss-of-control incident

UPDATE 3 July 2020: Flight International then followed with  Control-check ‘routine’ led E190 upset crew to miss reversed ailerons

UPDATE 2 November 2022: The Honourable Company of Air Pilots awarded the 2022 Hugh Gordon-Burge Memorial Award to the crew, Captain Vyacheslav Aushev, FO Sergey Sokolov, FO Bauyrzhan Karasholakov for “their outstanding behaviour and action throughout this incident, culminating in saving the lives of all onboard and the aircraft”.

Safety Resources

We also wrote Crossed Cables: Colgan Air B1900D N240CJ Maintenance Error.  In 2003 a B1900D crashed on take off after errors during flying control maintenance. We looked at the maintenance human factors safety lessons from this and another B1900 accident that year. We have previously discussed Misrigged Flying Controls: Fatal Maintenance Check Flight Accident, involving a Piper PA-46-350P Malibu Mirage (N962DA) in 2015, Too Rushed to Check: Misrigged Flying Controls involving a Piper PA-12 (N3280M) in 2017 and FAA Rules Applied: So Misrigged Flying Controls Undetected involving a Cessna 172M Skyhawk II (N9085H) in 2018.

We have also discuss in the past two cases were the normal sound airmanship of pre-flight ‘full and free’ control checks would have prevented an accident:

In 1992 the gust locks had not been fully disengaged on a third-party DHC-4T Turbine Caribou conversion, N400NC, leading to a horrific fatal accident that killed 3.

We have previously written:


FSF Maintenance Observation Programme (MOP)

Aerossurance worked with the Flight Safety Foundation (FSF) to create a Maintenance Observation Program (MOP) requirement for their contractible BARSOHO offshore helicopter Safety Performance Requirements to help learning about routine maintenance and then to initiate safety improvements: mop Aerossurance can provide practice guidance and specialist support to successfully implement a MOP. An excellent initiative Human Hazard Analysis (HHA) that could help identify problematic maintenance tasks and improvements is described in Designing out human error

HeliOffshore, the global safety-focused organisation for the offshore helicopter industry, is exploring a fresh approach to reducing safety risk from aircraft maintenance. Recent trials with Airbus Helicopters and HeliOne show that this new direction has promise. The approach is based on an analysis of the aircraft design to identify where ‘error proofing’ features or other mitigations are most needed to support the maintenance engineer during critical maintenance tasks.

Other manufacturers and MROs are expanding the use of HHA during 2019.

Aerossurance was pleased to both sponsor and present at a Royal Aeronautical Society (RAeSHuman Factors Group: Engineering seminar Maintenance Error: Are we learning? on 9 May 2019 at Cranfield University where we advocated a more sophisticated and holistic approach. raes are we learning

UPDATE 24 October 2022: The Royal Aeronautical Society (RAeS) has launched the Development of a Strategy to Enhance Human-Centred Design for MaintenanceAerossurance‘s Andy Evans  is pleased to have had the chance to participate in this initiative.

Aerossurance has extensive air safety, operations, airworthiness, human factors, aviation regulation and safety analysis experience.  For practical aviation advice you can trust, contact us at: enquiries@aerossurance.com