Singapore Airlines B777 41t Fuel Discrepancy Incident
The Singapore Transport Safety Investigation Bureau (TSIB) has released their safety investigation report into an incident that occurred on 16 April 2014 to Singapore Airlines Boeing B777-200ER 9V-SVC flying from Singapore to Johannesburg.
The Refuelling
Before departure the flight crew had requested the aircraft be fuelled to 86 tonnes. The Refuelling Dispenser Operator (RDO) proceeded to fuel the aircraft until the Lead Technician (LT) from the airline’s contracted maintenance organisation confirmed the aircraft’s total fuel had reached 86 tonnes. The LT was positioned under the aircraft wing at the aircraft’s Integrated Refuel Panel (IRP). The flight crew then confirmed that the figure tallied with the cockpit fuel indication.
Based on the refuelling dispenser’s fuel flow counter, the RDO presented to the LT a receipt for 121.5 tonnes of fuel uplifted to the aircraft.
Noticing the abnormally high fuel quantity stated on the RDO’s fuel receipt, the LT confirmed from the IRP that the final total fuel quantity was 86 tonnes. He informed the [Certifying Technician] CT of the fuel quantity discrepancy between the IRP and RDO figures.
The RDO showed the CT that the 121.5 tonnes measured by the fuel flow counter (i.e. fuel through the hose) matched the refuelling dispenser’s start/end quantity totaliser gauge (i.e. the fuel transferred out of the dispenser). The CT queried if the fuel flow counter had been reset before fuelling. The RDO was unsure and accepted the suggestion.
The CT informed the flight crew about the fuel discrepancy and that there would be a delay to departure. He then consulted his supervisor and was advised to perform a manual fuel quantity check for the aircraft’s fuel tanks using fuel measuring sticks [aka a ‘magnastick’].
Magnastick (Credit: TSIB)
Based on those results for the centre fuel tank and an assumption that the wing tanks were full, the CT arrived at a calculation of 86.001 tonnes, which confirmed the aircraft cockpit indication.
The aircraft departed after a 30 minute delay.
The Incident Flight
The TSIB say that the crew were…
…prompted with a “FUEL DISAGREE” aircraft system message one hour into the flight. The flight crew carried out the necessary checks and noticed that, at that point in the flight, the fuel quantity onboard as calculated by the Flight Management Computer (FMC) based on the quantity of the fuel that had burnt off during the flight was less than the fuel quantity as measured by the aircraft’s fuel quantity indicating system (FQIS), and the difference between these two quantities was increasing.
After consulting the airline’s maintenance centre, the flight crew decided to return to Singapore. The aircraft landed in Singapore without incident.
Investigations showed the aircraft had departed with approximately 127 tonnes of fuel, instead of 86 tonnes as displayed to the flight crew by the FQIS, a 41 tonne overload. The flight crew did not encounter any flight handling difficulties. The TSIB make no comment on the take off performance.
The B777 Fuel System
The Boeing 777-200 and 777-200ER (the Extended Range variant involved in this incident) have left and right wing tanks and a centre tank. The TSIB explain that:
The left and right wing tanks of both models had approximately similar capacity. However, the centre tank of Boeing 777-200 was much smaller than that of Boeing 777-200ER because the former included a large dry bay and the difference was about 40.5 tonnes.
The aircraft’s fuel quantity indicating system (FQIS) computed the total fuel quantity in the aircraft’s fuel tanks by way of a system of sensors in the fuel tanks (e.g. densitometers, water detectors, fuel temperature sensors) and sent the computed total fuel quantity figure to both the IRP and the cockpit for display.
Fuel tank sensors’ input and computation were handled by the FQIS’ Fuel Quantity Processing Unit (FQPU).
The bigger centre tank on the Boeing 777-200ER entailed an additional eight fuel sensors in the centre tank as compared to the Boeing 777-200. There was no difference between the two models in the fuel sensors arrangement for the wing tanks.
The aircraft was installed with a program switch module (PSM) consisting of pin switches that could be preset.
Boeing 777 Fuel Pin Switch Module (Credit: TSIB)
The PSM pin switch pattern for the Boeing 777-200ER was different from that for the Boeing 777-200 and allowed the FQPU to accept the inputs from the eight additional sensors… In other words, the PSM made the FQPU “know” which aircraft model it was dealing with so that the FQPU would get inputs from the correct sensors and compute the fuel quantity accordingly.
When Boeing examined the FQPU data after the incident they discovered he FQPU was operating in 200 mode, instead of -200ER mode. Consequently the inputs from the eight extra centre fuel tank sensors were not considered and the FQIS was under-reading the fuel quantity by around 41 tonnes. Boeing’s testing of the PSM could not identify any electrical or mechanical fault however, although some PSM switch wear was apparent.
The FQIS had also been operating normally prior to this incident flight and there was no recent maintenance work done to the FQIS or PSM.
The aircraft manufacturer indicated that it had received reports of FQIS faults in the past where the problems were identified to be with the FQPU, the PSM or poor connection between components. However, it did not identify any such problem on the FQIS components on this incident aircraft.
Analysis
With the defect in the aircraft systems, the next defence to a fuelling error was the dispensers gauges which did highlight a fuel anomaly. Verification was then dependent on the magnastick measurements and calculation. For the centre tank, the maintenance personnel had only taken magnastick readings from the No 1 position when they should have taken them from the No1 and No2 positions and so under read the contents of the centre tank.
The TSIB note that:
The CT and LT underwent training on refuelling operation which included theory on magnastick check. However, there were no practical session on magnastick check during their training on refuelling operation. In addition, magnastick check was not a commonly performed task in the course of their work
TSIB comment:
Practical training, initial and recurrent, on magnastick check must be made available to the technicians who need to perform this function. This is to ensure that they have the relevant experience and confidence in performing the task as well as the knowledge to compute the fuel quantity using the relevant refuelling manual.
It may be prudent to have another qualified person to conduct an independent magnastick check to verify the actual total fuel quantity in fuel tanks whenever a magnastick check of fuel quantity is required.
It would be desirable to have an aircraft safety system that could detect or alert the flight crew or maintenance crew when there was a mismatch between the aircraft model referenced by the FQIS and the actual aircraft model.
Safety Actions
TSIB report that:
- Immediately after the incident, the aircraft operator had replaced the incident aircraft’s FQIS components (e.g. the FQPU, the fuel tank sensors, the PSM and associated components).
- Following the incident, the aircraft maintenance service provider (employer of the CT and LT) implemented a one-time refuelling refresher training and competency assessments for various aircraft types for their personnel. Practical training on taking magnastick readings was also provided using a fuel tank simulator training tool.
- The AMSP conducted a briefing for their personnel on lessons learnt from this incident, and also reminded them to consult their managers whenever they are not familiar with any task assigned.
- The aircraft manufacturer upgraded subsequent versions of the FQPU to be able to detect and prevent incorrect program pins configuration.
- The aircraft manufacturer reviewed two areas of potential safety concerns pertaining to overfuelling, raised by the FQIS referencing an incorrect aircraft model, namely, runway overrun in a rejected take-off scenario and insufficient climb capability. The aircraft manufacturer determined that the aircraft would have sufficient safety margin in both scenarios and did not consider that the scenarios presented a safety hazard.
- In view of the actions in 4 and 5, the aircraft manufacturer is of the view that it is not necessary to put in place an aircraft safety system that could detect or alert the flight crew or maintenance crew when there is a mismatch between the aircraft model referenced by the FQIS and the actual aircraft mode.
TSIB Safety Recommendations
- The aircraft maintenance service provider require the magnastick check to be performed by two qualified personnel independently. [TSIB Recommendation RA-2018-012]
- The civil aviation authority consider requiring the aircraft maintenance service providers to have magnastick checks performed by two qualified personnel independently. [TSIB Recommendation RA-2018-013]
- The aircraft maintenance service provider consider the need for refresher training on magnastick check to be conducted for personnel at a suitable time interval. [TSIB Recommendation RA-2018-014]
Unrelated Development (ADDED 6 June 2020)
Boeing is developing a modification after the discovery of a 777 centre wing tank fuel indication problem. It:
…follows at least 25 instances of disparity between the aircraft’s fuel-quantity indicator for the centre tank and the uplifted amount from refuelling trucks – after which the crew received a ‘fuel disagree’ or ‘insufficient fuel’ message within 3h of departure.
At least 16 of these occurrences led the crew to continue the flight, after which six aircraft arrived at their destination while 10 had to land at a diversion airport.
UPDATE 22 March 2022: This eventually resulted in a proposed FAA Airworthiness Directive (AD) to correct the design flaw.
Other Safety Resources
- Aerossurance has written about a very different B777 fuel problem in: Forgotten Fasteners – Serious Incidents
- Investigation into Jet Airways B777 VT-JEK Serious Incident at Heathrow
- Asiana B777 Accident at SFO 6 July 2013
- B777 in Autoland Mode Left Runway When Another Aircraft Interfered With the Localiser Signal
- Lufthansa MD-11F Nose Wheel Detached after Maintenance Error
- UPDATE 6 February 2020: Emirates B777 Runway Impact During Attempted Go-Around, 3 August 2016, Dubai: Accident Report
We have also published:
- Airworthiness Matters: Next Generation Maintenance Human Factors Over the last 10-15 years, much attention has been focused on reporting & investigating errors. However, this can not not prevent events involving tasks rarely conducted until after a potential catastrophic error.
- Aircraft Maintenance: Going for Gold? Should we start treating maintenance personnel more like athletes who need to achieve peak performance every day?
- The Power of Safety Leadership
- Leadership and Trust
- How To Develop Your Organisation’s Safety Culture
- James Reason’s 12 Principles of Error Management
- Back to the Future: Error Management
Aerossurance worked with the Flight Safety Foundation (FSF) to create a Maintenance Observation Program (MOP) requirement for their contractible BARSOHO offshore helicopter Safety Performance Requirements to help learning about routine maintenance and then to initiate safety improvements:
Aerossurance can provide practice guidance and specialist support to successfully implement a MOP.
UPDATE 24 June 2018: B1900D Emergency Landing: Maintenance Standards & Practices The TSB report posses many questions on the management and oversight of aircraft maintenance, competency and maintenance standards & practices. We look opportunities for forward thinking MROs to improve their maintenance standards and practices.
UPDATE 31 May 2019: The Portuguese accident investigation agency, GPIAAF, issued a safety investigation update on a serious in-flight loss of control incident involving Air Astana Embraer ERJ-190 P4-KCJ that occurred on 11 November 2018. The aircraft was landed safely after considerable difficulty, so much so the crew had debated ditching offshore. GPIAAF conformed that incorrect ailerons control cable system installation had occurred in both wings during a maintenance check conducted in Portugal.
Misrigged Flying Control Cable, Air Astana Embraer EJ-190 P4-KCJ (Credit: GPIAFF)
GPIAFF note that: “By introducing the modification iaw Service Bulletin 190-57-0038 during the maintenance activities, there was no longer the cable routing and separation around rib 21, making it harder to understand the maintenance instructions, with recognized opportunities for improvement in the maintenance actions interpretation”. They also comment that: “The message “FLT CTRL NO DISPATCH” was generated during the maintenance activities, which in turn originated additional troubleshooting activities by the maintenance service provider, supported by the aircraft manufacturer. These activities, which lasted for 11 days, did not identify the ailerons’ cables reversal, nor was this correlated to the “FLT CTRL NO DISPATCH” message.”
GPIAFF comment “deviations to the internal procedures” occurred within the maintenance organisation that “led to the error not being detected in the various safety barriers designed” in the process. They also note that the error ” was not identified in the aircraft operational checks (flight controls check) by the operator’s crew.”
UPDATE 1 June 2019: Our analysis: ERJ-190 Flying Control Rigging Error
UPDATE 25 June 2020: Japanese Jetstar Boeing 787 GEnx-1B Engine Biocide Serious Incident
Recent Comments