Crossed Cables: Colgan Air B1900D N240CJ Maintenance Error
On 26 August 2003, Beechcraft 1900D, N240CJ, operated by Colgan Air (operated for US Airways Express) crashed shortly after take off on a post-maintenance ferry flight from Barnstable Municipal Airport, Hyannis, Massachusetts, to Albany International Airport, New York.
The Accident Flight
During the takeoff roll, the pitch trim system began to move in the downward direction. The initial movement from 1.5º to 3º nose down was consistent with an electric pitch trim motor input. Four seconds later the pitch trim movement went from 3º to 7º nose down faster than the trim motor could achieve. The US National Transportation Safety Board (NTSB) explain in their final report that:
The digital flight data recorded (DFDR) indicated that shortly after declaring an emergency, the airplane began a left turn while climbing to 1,100 feet. Engine torque was reduced, and the airplane remained at 1,100 feet while maintaining an airspeed of approximately 207 knots and 30 degrees of left bank for 15 seconds. The airplane then pitched down to 8 degrees negative (nose down) and the airspeed increased to 218 knots. The airplane rolled right and left due to control inputs, and the pitch attitude decreased to 30 degrees negative.
The crew declared an emergency, reporting runway trim, and attempted to return to the airport. The investigation showed that the flight crew had manually selected nose up trim settings, but the aircraft trimmed full nose down. The control column force reached 250 pounds but the crew was unable to maintain control and the aircraft impacted the water off Yarmouth, Massachusetts nose first. Both flight crew, the only occupants, were killed.
The accident aircraft had been brought in for its Detail 6 (D6) check on Saturday 23 August 2003, which commenced at 16:47. The D6 is one a series of 6 zonal checks conducted over 1320 flying hours. The emphasis of this check is the empennage and aft fuselage, engine borescope inspections, engine mount torque checks, operational checks and flying control cable tension checks.
On the following morning the check was interrupted at 08:00 and the remaining work was deferred. Ten revenue flights were then conducted. At 20:30 on the evening of 24 August 2003 the aircraft was returned for completion of the D6 check which was concluded at 11:00 26 August 2003. The NTSB explain that:
A maintenance technician conducted a free play check of the left and right elevator trim actuators as part of the Detail Six phase check. Both actuators failed the check, and the failure required replacement of the actuators.
During the replacement of the actuators, the technician explained in interview with the NTSB that they chose to omit AMM steps (c) of the Elevator Trim Tab Actuator Removal and step (i) of the Elevator Trim Tab Actuator Installation. Omission of these steps also eliminated 19 referenced AMM steps. The technician stated that that was how he had been trained to do the task and Colgan explained that this deviation was rationalised because:
In the mind of the Colgan mechanics it was thought that they actually performed the R[emoval] & R[e-installation] of the elevator trim tab actuators in a safe manner because they avoided having to disengage or remove major elements of the flight control system, and disrupted a much smaller portion of the airplane.
The technician also did not maintain pressure on the elevator trim tab cables (known as blocking), although the AMM does not require that the cables be blocked.
Subsequently, the cable unwound off the forward drum. On 25 August, during the operational check of the system, the forward elevator trim tab cable “fell off” the forward drum, seized, and kinked.
Colgan told the NTSB that feedback from the aircraft manufacturer suggested this was a known problem.
A new forward elevator trim tab cable was ordered. Due to an incorrect right elevator trim actuator part number [having been installed earlier in the check], a new right elevator trim actuator was also ordered.
Colgan claim the incorrect actuator had previously been fitted after initial consultation with the aircraft manufacturer’s tech rep, as the Illustrated Parts Catalogue (IPC) did not list any available part numbers.
That evening, two lead maintenance technicians replaced the forward elevator trim tab cable, and two other maintenance technicians replaced the right elevator trim actuator. The forward elevator trim tab cable drum had already been removed by personnel on the dayshift, but no turnover [i.e. handover] notes were forwarded. The AMM and Colgan Air policies did not require turnover notes from one shift to another.
The two lead maintenance technicians that replaced the forward elevator trim tab cable did not use a lead wire as instructed by the AMM.
Lead wires are used to assure the proper routing of cables and electrical wiring when being removed during maintenance. A thin wire or string is attached to the cable and “follows” the cable as it is pulled out of the pulley system (electrical wire out of the conduit). The lead wire becomes a visual guide for the removed cable’s route and is attached to the replacement cable. It is then pulled back through the system to assure the replacement is properly routed back.
They stated that they felt that this step was unnecessary. They marked the topmost cable pulleys with a “T” instead to keep track of which pulley the left hand threaded cable should go through.
Despite these deviations…
The two lead maintenance technicians…stated that they referred to the AMM…
The NTSB observed that AMM Chapter 27-30-04, “Elevator Trim Tab Cables – Maintenance Practices” depicted the trim drum back to front (i.e. showing the open, keyed side of the drum instead of the flat side of the drum).
While the NTSB note that the drum could not in practice be installed as illustrated, i.e. backwards, “it was possible to mis-route the cable around the drum, and reverse the trim system”.
The depiction in the maintenance manual showed the nose-up trim tab cable emanating from the aft end of the drum, rather than the forward end. It also showed the nose-down cable emanating from the forward end of the drum, rather than the aft. However, the “FORWARD AS INSTALLED” arrow included in the depiction would have to be ignored, and the cables would have to be crossed once along the cable run, to reverse the system and secure the cable ends into the turnbuckles.
This illustration has been in the AMM since 1984, with no prior report of problems according to the manufacturer (though they do not explain how they themselves failed to spot the error). The technicians stated that they were not confused handling the drum or interpreting the drum illustration.
The Safety Investigation
Upon examination of the wreckage:
The right and left elevator trim actuators were found near the full nose-down elevator trim position [but]…examination of the manual trim wheel revealed that it was found near the 6.5 units of nose-up trim position.
The evidence of misleading maintenance instructions for rigging the forward cable drum, and the FDR data which showed the elevator trim moving in the nose down direction, led the [NTSB] Airworthiness Group to investigate the possibility of incorrectly rigging the elevator trim system.
The NTSB performed a misrigging demonstration on another B1900:
When the cockpit trim wheel was positioned nose down, the elevator trim tabs moved in a nose-up direction. When the cockpit trim wheel was positioned nose-up, the elevator trim tabs moved in a nose-down direction. When the electric trim motor was activated in one direction, the elevator tabs moved in the corresponding correct direction, but the trim wheel moved opposite of the commanded electric trim direction.
The mis-rigging demonstration also revealed that when the manual trim wheel was in the nose-down direction, the trim indicator in the cockpit moved well past the nose down limit, and the trim tabs were in the full nose up position. When the manual trim wheel was moved in the nose-up direction, the trim indicator did not reach the nose up limit. Rather, the indicator stopped near positive “3″ units, and the trim tabs were in the full nose down position.
A lead maintenance technician and the quality assurance inspector stated that “a successful operational check of the system was completed”. NTSB note that…
…there was no procedure for an operational check contained in [AMM] Chapter 27-30-04.
The maintenance personnel say that…
…testing of the system was based on what they determined to be proper tests for properly testing the system components.
It therefore appears to have been an undocumented test they themselves created. An inspector then did further checks, again without reference to a procedure, in order to sign a the Required Independent Inspection (RII).
Colgan note that:
One item of apparent focus in the oral interviews of the mechanics was the fact that the maintenance, and the Detail 6 inspection/check, was performed by a crew working the night shift. All of the men interviewed stated that they were not fatigued, were not ill, were not experiencing any undue stress at work or at home, and had no difficulty obtaining rest through sleep during their non-working hours. They were not rushed to complete the job.
The NTSB found that the flight crew did not perform a first flight of the day check, as company policy and manuals required, and so missed a further chance to detect the problem.
Using an FAA certified Level “D” Beech 1900 full motion simulator, the [NTSB Operations] Group attempted six simulations of the accident flight. During all simulations, the elevator trim was positioned full nose-down shortly after takeoff. The simulator pilot attempted to maintain aircraft control using different power settings to obtain different airspeeds. Five of the six simulations resulted in an uncontrolled descent into terrain. On the sixth test, the simulator pilot was able to partially maintain control of the airplane by gradually reducing engine power and maintaining an airspeed of approximately 170 knots. However, he had to return to the airport area at 170 knots, and touchdown at 180 knots. The airplane did not land on a runway, and subsequently impacted terrain.
NTSB Probable Cause
The NTSB determined that the probable cause to be:
The improper replacement of the forward elevator trim cable, and subsequent inadequate functional check of the maintenance performed, which resulted in a reversal of the elevator trim system and a loss of control in-flight.
Contributory factors were:
- the flightcrew’s failure to follow the checklist procedures, and
- the aircraft manufacturer’s erroneous depiction of the elevator trim drum in the maintenance manual.
During the course of the investigation:
- Temporary Revision 27-9 of the AMM was issued 12 September 2003 to add a Manual Elevator Trip Operational Check followed by Safety Communiqué 234 on 24 September 2003
- Temporary Revision 27-10 was issued 22 October 2003, which revised AMM 27-30-04 Elevator Trim Tab Cable Installation and updated the depiction of the forward trim drum.
- Four further Temporary Revisions were issued between December 2013 and March 2004 on relevant aspects of the flying controls.
- The FAA issued Airworthiness Directive (AD2003-20-10), which instructed operators to incorporate TR-27-9, and provided a change to the maintenance illustration depicting the forward trim drum (see below).
- Colgan Air issued an alert to its employees regarding possible trim problems and expanded the trim check procedure on the First Flight of the Day and the Taxi checklists.
An Earlier Accident: B1900D N233YV, Air Midwest Flight 5481, Charlotte, 8 January 2003
Just a few months before there was another catastrophic B1900D loss of control accident with some common features that one might have expected would have heighten attention. The report on this accident included several safety recommendations relevant to the Colgan accident at Yarmouth.
On 8 January 2003, N233YV, flying Air Midwest Flight 5481, also operated for US Airways Express, crashed into a hangar shortly after takeoff from Charlotte, North Carolina. Both flight crew members and all 19 passengers were killed.
The NTSB report subsequently established that after takeoff, the pilots had also been unable to control the pitch of the aircraft.
This was the aircraft’s tenth flight after a D6 check by a 3rd party maintenance organisation. The previous nine flights all involved fewer passenger loads and a more forward CG. The loss of control on the 10th flight was because:
- The aircraft was overloaded by 580 lb and had an aft centre of gravity that was 5% beyond limits.
- The elevator control system did not have the full range of nose-down travel, due to incorrect rigging that had occurred during a maintenance visit just over 24 hours prior to the accident.
On the night of 6-7 January 2003, the aircraft had undergone a scheduled maintenance check that included checking the tension of the elevator control cables.
The technician involved was performing this task for the first time and so was receiving on-the-job training from a inspector, despite the clear conflict of interest that the inspector would do the Required Independent Inspection (RII) on the task performed by the technician under his guidance.
The inspector stated that he did not think he needed to closely supervise the technician because of he had previous flight control rigging experience. A relatively high proportion of personnel in the facility were under going On the Job Training (OJT). The FAA had identified “serious deficiencies” in the training procedures at the facility, but these had not been corrected.
Finding that the cable tension was low, the technician performed only selected steps from the comprehensive elevator control system rigging procedure to increase tension using cable turnbuckles. Both the technician and inspector were under the misapprehension that the whole rigging procedure was not needed if merely tensioning the cables.
However, the technician inadvertently restricted the amount of nose down elevator travel that was possible to about half of what should have been available. A properly conducted functional test should have caught this error, but one was not required at the time.
The NTSB determined the Probable Cause of the Charlotte accident to be:
The airplane’s loss of pitch control during takeoff. The loss of pitch control resulted from the incorrect rigging of the elevator control system compounded by the airplane’s aft center of gravity, which was substantially aft of the certified aft limit.
Contributing factors were:
- Air Midwest’s lack of oversight of the work being performed at the Huntington, West Virginia, maintenance station
- Air Midwest’s maintenance procedures and documentation
- Air Midwest’s weight and balance program at the time of the accident
- the inspector’s failure to detect the incorrect rigging of the elevator system
- the FAA’s average weight assumptions in its weight and balance program guidance at the time of the accident
- the FAA’s lack of oversight of Air Midwest’s maintenance program and its weight and balance program.
Among the safety recommendations were:
- Safety Recommendation A-04-004 to the FAA to perform “targeted surveillance and increased oversight of maintenance practices”. The FAA confirmed “the need for FAA maintenance inspectors to observe actual maintenance work being performed and to confirm that the correct procedures are being used”.
- Safety Recommendation A-04-006 to the FAA “to require that the Instructions for Continued Airworthiness and air carrier maintenance manuals…include a complete functional check at the end of maintenance for each critical flight system”. The FAA instead issued Order 8110.54, Instructions for Continued Airworthiness Responsibilities, Requirements and Contents in July 2005.
- Safety Recommendation A-04-007 to the FAA to “require manufacturers of aircraft operated under 14 Code of Federal Regulations Part 121 to identify appropriate procedures for a complete functional check of each critical flight system; determine which maintenance procedures should be followed by such functional checks; and modify their existing maintenance manuals, if necessary, so that they contain procedures at the end of maintenance for a complete functional check of each critical flight system”. The FAA responded to this 12 years later with Policy Statement PS-ANM-25-18, Post Maintenance Checks and Tests a response NTSB classified in 2017 as unacceptable as it only applies to future certifications.
- Safety Recommendation A-04-008 to the FAA to “require…Part 121 air carriers to modify their existing maintenance manuals, if necessary, so that they contain procedures at the end of maintenance for a complete functional check of each critical flight system. ” Both FAA and NTSB agree that this action is still open, but NTSB believe the FAA plan is unacceptable.
- Safety Recommendation A-04-009 to the FAA to “prohibit inspectors from performing required inspection item inspections on any maintenance task for which the inspector provided on-the-job training to the mechanic who accomplished the task.” This finding was issued because in September 2003 the FAA notified the inspector involved in this accident that FAA enforcement investigation “did not establish a violation of the Federal Aviation Regulations.” In 2010 the FAA issued clarification that inspecting work performed by a trainee you supervised is prohibited.
- Safety Recommendation A-04-010 to the FAA to “require…Part 121 air carriers that use contractors to perform required inspection item (RII) maintenance tasks and inspections to have air carrier personnel who are physically present when a substantial amount of the RII planning, tasking, maintenance work, and inspections are performed and are readily available when they are not physically present and who ensure that the processes and procedures used by contractors to perform RII maintenance tasks and inspections are the same as those used by air carrier maintenance personnel”. This was closed after the October 2010, revision to FAA Order 8900.1 which added a new subsection 3-3886, paragraph P, recommending that “the certificate holder assign a qualified and authorized employee at the inspection site to oversee the performance of required inspections by other persons and, in cases when the assigned air carrier employee cannot be physically present, that there be a method of contacting that employee”.
- Safety Recommendation A-04-011 to the FAA to “develop detailed on-the-job (OJT) training requirements for…Part 121 air carriers…”. This remains open in 2019.
- Safety Recommendation A-04-012 to the FAA to “audit training records for personnel who are currently performing maintenance on Air Midwest airplanes…”. This was completed one year but “the NTSB is disappointed that the FAA did not inform the NTSB of this fact for nearly 4 years”!
- Safety Recommendation A-04-013 to the FAA to “require…Part 121 air carriers to implement a program in which carriers and aircraft manufacturers review all work card and maintenance manual instructions for critical flight systems and ensure the accuracy and usability of these instructions so that they are appropriate to the level of training of the mechanics performing the work”. This was closed but based on what the NTSB considered an “unacceptable response” as NTSB disagreed with the FAA’s new standardised definition of essential maintenance. They noted that “an FAA-sponsored survey of aviation maintenance technical manuals reported that manufacturers commonly made reactive rather than proactive use of user evaluations, made limited use of user input and procedure validation, did not attempt to systematically track errors, and lacked standards for measuring document quality.”
- Safety Recommendation A-04-015 to the FAA to “require…Part 121 air carrier maintenance training programs be approved. The FAA reported that the “Air Carrier Maintenance Training Programs rulemaking project was cancelled in August 2015 due to a lack of accident data to support rulemaking activities”. When that was initiated in 2011 it was believe it was relevant to 35 past accidents. It was cancelled when this assessment reduced to two (non-Part 121 accidents). The action has since been closed on the basis that FAA oversight does include training programmes.
- Safety Recommendation A-04-016 to the FAA to “require…Part 121 air carriers implement comprehensive human factors programs to reduce the likelihood of human error in aviation maintenance”. This remains open, even though Europe first implemented rule making on this topic in 2003 with JAR-145 Amendment 5 (ironically the same year as both these accidents). This remains open in 2019.
- Safety Recommendation A-04-024 to the FAA to ensure the manufacturer “revises the maintenance procedures for critical flight systems in its Beech 1900, 1900C, and 1900D Airliner Maintenance Manuals to ensure that the procedures can be completely and correctly accomplished“. These revisions were done by 2005.
Aerossurance has previously written on these associated topics:
- Fatal $16 Million Maintenance Errors: Airbus Helicopters AS350B2 N37SH, operated by Sundance Helicopters, suffered a Loss of Control (LOC) accident in mountainous terrain east of Las Vegas on 7 December 2011. This accident pointed to a number of serious maintenance issues, and comments on maintenance human factors that stretch back to the 2003 B1900D accidents.
- Airworthiness Matters: Next Generation Maintenance Human Factors: Over the last 10-15 years, much attention has been focused on maintenance human factors training and reporting & investigating errors. While we could concentrate on simply doing more of these and certainly can find ways to do these things better, perhaps the next generation approach needs to include a much wider range of activities.
- Aircraft Maintenance: Going for Gold?: Should we start treating maintenance personnel more like athletes who need to achieve peak performance every day?
- The Power of Safety Leadership
- Leadership and Trust
- Safety Performance Listening and Learning – AEROSPACE March 2017
- How To Develop Your Organisation’s Safety Culture
- James Reason’s 12 Principles of Error Management
- Back to the Future: Error Management
- Critical Maintenance Tasks: EASA Part-M & -145 Change
- B747 Landing Gear Failure Due to Omission of Rig Pin During Maintenance
- When Down Is Up: 747 Actuator Installation Incident
- Lost in Translation: Misrigged Main Landing Gear
- Maintenance Human Factors in Finnish F406 Landing Gear Collapse Safety investigators discuss the associated maintenance human factors and inadequacies in the type’s maintenance instructions.
- Hoist Assembly Errors: SAR Personnel Dropped Into Sea Taiwanese investigators determined that the hoist hook had been inappropriately assembled during maintenance 2 days earlier, raising questions on the technical knowledge of hoist maintenance.
- Misassembled Anti-Torque Pedals Cause EC135 Accident
- EC130B4 Accident: Incorrect TRDS Bearing Installation and
- Misrigged Flying Controls: Fatal Maintenance Check Flight Accident
- Too Rushed to Check: Misrigged Flying Controls
- Fire After O-Ring Nipped on Installation
- B1900D Emergency Landing: Maintenance Standards & Practices The TSB report posses many questions on the management and oversight of aircraft maintenance, competency and maintenance standards & practices. We look at opportunities for forward thinking MROs to improve their maintenance standards and practices.
- UPDATE 13 November 2018: Inadequate Maintenance, An Engine Failure and Mishandling: Crash of a USAF WC-130H: investigators discuss a strong cultural overtone in this accident that killed 9.
- UPDATE 18 December 2018: USAF Engine Shop in “Disarray” with a “Method of the Madness”: F-16CM Engine Fire
- UPDATE 9 February 2019: Meeting Your Waterloo: Competence Assessment and Remembering the Lessons of Past Accidents: No one was injured in this low speed derailment in London after signal maintenance errors but investigators expressed concern that the lessons about maintenance errors from the fatal triple collision at Clapham in 1988 may have been forgotten.
- UPDATE 30 March 2019: Contaminated Oxygen on ‘Air Force One’ Poor standards at a Boeing maintenance facility resulted in contamination of two oxygen systems on a USAF Presidential VC-25 (B747).
- UPDATE 19 April 2019: FAA Rules Applied: So Misrigged Flying Controls Undetected in an accident to a Cessna 172 in Bermuda.
In particular we like this free CASA human factors video: Crossed Wires: Online Maintenance HF Training Video
UPDATE 31 August 2018: A more recent flying control cable problem: Maintenance Mistakes May Have Caused 2017 US Navy MH-60R Helicopter Crash off Guam
UPDATE 31 May 2019: The Portuguese accident investigation agency, GPIAAF, issued a safety investigation update on a serious in-flight loss of control incident involving Air Astana Embraer ERJ-190 P4-KCJ that occurred on 11 November 2018. The aircraft was landed safely after considerable difficulty, so much so the crew had debated ditching offshore. GPIAAF conformed that incorrect ailerons control cable system installation had occurred in both wings during a maintenance check conducted in Portugal.
GPIAFF note that: “By introducing the modification iaw Service Bulletin 190-57-0038 during the maintenance activities, there was no longer the cable routing and separation around rib 21, making it harder to understand the maintenance instructions, with recognized opportunities for improvement in the maintenance actions interpretation”. They also comment that: “The message “FLT CTRL NO DISPATCH” was generated during the maintenance activities, which in turn originated additional troubleshooting activities by the maintenance service provider, supported by the aircraft manufacturer. These activities, which lasted for 11 days, did not identify the ailerons’ cables reversal, nor was this correlated to the “FLT CTRL NO DISPATCH” message.”
GPIAFF comment “deviations to the internal procedures” occurred within the maintenance organisation that “led to the error not being detected in the various safety barriers designed” in the process. They also note that the error ” was not identified in the aircraft operational checks (flight controls check) by the operator’s crew.”
UPDATE 1 June 2019: Our analysis: ERJ-190 Flying Control Rigging Error
An excellent initiative Human Hazard Analysis (HHA) that could help identify problematic maintenance tasks and improvements is described in Designing out human error
HeliOffshore, the global safety-focused organisation for the offshore helicopter industry, is exploring a fresh approach to reducing safety risk from aircraft maintenance. Recent trials with Airbus Helicopters and HeliOne show that this new direction has promise. The approach is based on an analysis of the aircraft design to identify where ‘error proofing’ features or other mitigations are most needed to support the maintenance engineer during critical maintenance tasks.
Other manufacturers and MROs are expanding the use of HHA during 2018.
Aerossurance worked with the Flight Safety Foundation (FSF) to create a Maintenance Observation Program (MOP) requirement for their contractible BARSOHO offshore helicopter Safety Performance Requirements to help learning about routine maintenance and then to initiate safety improvements:
Aerossurance and its partners can provide proven, practical expertise to successfully implement a MOP, introduce an effective competence assessment process or analyse the hazards of maintenance tasks.
Aerossurance is pleased to be both sponsoring and presenting at a Royal Aeronautical Society (RAeS) Human Factors Group: Engineering seminar Maintenance Error: Are we learning? to be held on 9 May 2019 at Cranfield University.