When Humans and Robots Communicate

…they do not always achieve mutual understanding. And when misunderstandings do occur, the consequences can range from irritating to expensive to tragic.

On July 6 of 2013, Asiana Airlines Flight 214 crashed on final approach to San Francisco International Airport, resulting in over 180 injuries, 3 fatalities, and the loss of the aircraft. While the NTSB report on this accident is not yet out, there are several things that seem to be pretty clear:

–The flight crew believed that airspeed was being controlled by the autothrottle system, a device somewhat analogous to the cruise control of an automobile

–In actuality, the airspeed was not being controlled by the autothrottles

–The airspeed fell below the appropriate value, and the airplane dipped below the proper glidepath and mushed into the seawall

It is not yet totally clear why the autothrottle system was not controlling the airspeed when the captain and first officer believed that it was doing so. It is possible that the autothrottle mechanism failed, even that it failed in such a way that its failure was not annunciated. It is possible that an autothrottle disconnect button (one on each power level) was inadvertently pressed and the disconnection not noticed. But what seems likely in the opinion of several knowledgeable observers is that the captain and FO selected a combination of control settings that they believed would cause the autothrottle to take control–but that this setting was in fact not one that would cause autothrottle activation…in other words, that the model of aircraft systems in the minds of the flight crew was different from the actual design model of the autothrottle and its related systems.

Whatever happened in the case of Asiana Flight 214…and all opinions about what happened with the autothrottles must be regarded as only speculative at this point…there have been numerous cases–in aviation, in medical equipment, and in the maritime industry–in which an automated control system and its human users interacted in a way that either did or could have led to very malign results. In his book Taming HAL, Asaf Degani describes several such cases, and searches for general patterns and for approaches to minimize such occurrences in the future.

Degani discusses human interface problems that he has observed in common consumer devices such as clocks, TV remote controls, and VCRs, and goes into depth on several incidents involving safety-critical interface failures. Some of these were:

The airplane that broke the speed limit. This was another autothrottle-related incident, albeit one in which the consequences were much less severe than Asiana 214. The airplane was climbing to its initial assigned altitude of 11,000 feet, under an autopilot mode (Vertical Navigation) in which speed was calculated by the flight management system for optimum efficiency–in this case, 300 knots.  Air traffic control then directed that the flight slow to 240 knots for separation from traffic ahead. The copilot dialed this number into the flight control panel,overriding the FMS-calculated number.  At 11000 feet, the autopilot leveled the plane, switched itself into ALTITUDE HOLD mode, and maintained the 240 knot speed setting. Everything was fine.

The controller then directed a further climb to 14000 feet. The copilot re-engaged VERTICAL NAVIGATION MODE and put in the new altitude setting. The engines increased power, the nose pitched up, and the airplane began to climb. But just a little bit later, the captain observed that the airplane wasn’t only climbing–it was also speeding up, and had reached almost 300 knots, thereby violating an ATC speed restriction.

What happened here? Degani refers to events of this sort as “automation surprises.” The copilot was apparently thinking that the speed he had dialed in to override the flight management system would continue to be in force when he re-enabled the vertical navigation climb mode. But that wasn’t the way the system was actually designed. Selecting Vertical Navigation mode re-initialized the source of the airspeed command to be the FMS, which was still calling for a 300-knot Best Efficiency speed.

Degani says that the pilots were well trained and understood how the speed reference value actually worked…but that the unintuitive nature of the interface caused this knowledge to be effectively forgotten at the moment when the additional climb was requested. He draws an analogy with the user of a cordless phone, who picks up the ringing phone and pushes the TALK button..a seemingly-logical action that actually turns off the phone and disconnects whoever is calling.

The blood-pressure monitor that didn’t monitor.  A surgery patient was under anesthesia; as is standard practice, his blood pressure was being monitored by an electronic device. The patent’s blood pressure showed a high reading, and the surgeon noted profuse bleeding. The anesthesiologists set the blood-pressure monitor to measure more frequently. Periodically, they glanced back at the monitor’s display, noting that it still showed an elevated blood pressure, actively treating the hypertension–as they believed it was–with drugs that dilated blood vessels.

But actually, the patient’s blood pressure was very low. The alarmingly-high blood pressure values shown in the display were actually constant…the machine was displaying the exact same value every time they looked at it, because after the measurement-interval reset, it had never made another measurement.

What happened here? The blood-pressure monitor has three modes: MANUAL (in which the pressure is measured immediately when the “start” button is pressed), AUTOMATIC (in which pressure is measured repeatedly at the selected interval), and IDLE. When the interval is changed by the anesthesiologist, the mode is set at IDLE, even if the monitor were already running in AUTOMATIC.  To actually cause the automatic monitoring to occur, it is necessary to push START. In this case, the pushing of the START button was omitted, and the machine’s display did not provide adequate cues for the anesthesiologists to notice their mistake.

Critiquing the machine’s design, Degani notes that “The kind of change they sought is not very different from changing the temperature setting in your toaster over…On almost every oven, you simply grab the temperature knob and rotate it from 300 Farenheit to 450, and that’s it. You are not expected to tell the system that you want it to stay in OVEN mode–you know that it will.”

 

The ship that got confused. How does a ship run aground when it is equipped with GPS and LORAN and radar and a depth sounder…and operating in well-marked and charted coastal waters? This actually happened to the cruise ship Royal Majesty.

The first event in the chain leading to this grounding was that the antenna cable for the GPS came loose. The GPS, now with no source of true position from the satellites, defaulted as designed into Dead Reckoning mode, in which changes in position are calculated based on compass heading and ship’s speed. A DR position will initially be quite accurate, but the accuracy will degrade over time since there is no way to compensate for the effects of current, wind, and waves.

Dead reckoning is a perfectly respectable means of navigation, and many successful voyages have been completed with its assistance. But the officers of those ships knew they were using DR, and hence could make allowances for the inherent inaccuracy in their estimated positions. The officers of the Royal Majesty had no such knowledge.

When the ship’s GPS unit lost its signal and went into DR mode, it chirped an alarm…which nobody heard…and displayed the messages  dr and sol (for solution) on its panel, along with the calculated and changing latitude and longitude, which were in much larger type. The crew continued manually plotting the displayed latitude and longitude, and no one noticed the unobtrusive system status messages.

In addition to the manual plotting, the position from the GPS drove a moving-map display, which was overlayed on the radar scope. Position updates from the GPS to the map unit were sent periodically, and when the GPS went into Dead Reckoning mode, it flagged those messages accordingly…instead of sending a message like “GPS, 12:10:34, latitude, longitude, valid”, it sent “GPS, 1:11:46, latitude, longitude, invalid.” The assumption was that whatever device was receiving these message would be programmed to look at the valid/invalid flag, and to notify the operator to use caution in the latter case…because the accuracy of the position would be continuously degrading. The radar and map system on this ship, however, was not programmed to look at the relevant part of the message. Hence, the people doing the manual plotting did not notice the tiny messages warning that DR mode had been entered, and the people looking at the radar/map unit were given no indication whatsoever that anything was abnormal.

28 hours after the loss of the GPS signal, the Royal Majesty was 14 miles southwest of her intended route. At 6:45, the chief officer saw a blip on the radar, 7 nautical miles away from the ship. The blip was colocated on the map with the BA buoy, which marked the entrance into the Boston traffic lanes. All seemed well. But actually, the buoy that was displayed on the radar–and was visually sighted though not properly identified–was not the BA buoy, but rather a totally different marker known as the Asia Rip buoy.

Degani summarizes:

As the gray sky turned black veil, the phosphorus-lit radar map with its neat lines and digital indication seemed clearer and more inviting than the dark world outside. As part of a sophisticated integrated bridge system, the radar map had everything–from a crisp radar picture, to ship position, buoy renderings, and up to the last bit of data anyone could want–until it seemed that the entire world lived and moved transparently, inside that little green screen. Using this compelling display, the second officer was piloting a phantom ship on an electronic lie, and nobody called the bluff.

The bluff was finally called by reality itself, at 10 PM, when the ship jerked to the left with a grinding noise. The captain ran to the radar map, extended the radar range setting to 24 miles, and saw an unmistakable crescent-shaped mass of land: Nantucket Island. “Hard right,” he ordered.

But it was too late. The ship hit the Rose and Crown Shoal, hard, and could not be backed off. “He (the captain) couldn’t fathom how the ocean had betrayed him”…he ran to the chart table and plotted the position from the LORAN-C receiver, which was entirely independent of the GPS. The true position of the ship now differed by 17 miles from the GPS-derived position that the officers has been assuming was correct.

No one was injured, but repairs to the hull damage were estimated at $2 million.

Diverging lines never intersect.  In 1983, Korean Airline flight 007…far off-course…entered Soviet airspace and was shot down, resulting in the loss of all 269 passengers and crew aboard. Many theories have been promulgated about why this plane was so far away from where it was supposed to be: Degani places the blame firmly on the human-automation interface.

While still over Alaska, the flight crew put the autopilot in HEADING mode, with a selected compass heading of 245 degrees. Their intent was to intercept the international airway R-20, with the autopilot then following this track under the direction of the aircraft’s inertial navigation system. Degani believes that the crew, after becoming established on the 245 degree heading, probably switched the autopilot to INS (Inertial Navigation System) mode.

But the way this autopilot actually worked was that selecting INS would only arm the intertial mode: the airplane would not actually transition from compass heading to inertial control until it was within a specified distance (7.5 nautical miles) of the desired inertial course. This makes sense when you want to fly a particular heading to reach the inertial course, and not actually turn to that new course until you intercept it, or at any rate get close to it.

But a course of 245 degrees, from the position the airplane was evidently at when it was selected will never intercept the R-20 track. According to Degani, the inertial mode remained ARMED..but never engaged..throughout the entire remainder of the flight. The inertial navigation system itself did have an indicator that would show the pilot the active mode: the letters INS appeared in amber when the system was armed, green when it was engaged. But the autopilot itself had no indicator showing that the rotary switch setting to INS had not yet taken effect–and on this particular flight, would never take effect.

Other factors contributed to the tragedy: the presence of a military RC-135 plane in the vicinity, which increased Soviet suspiciousness, the darkness, which made visual identification impossible, the fact that the Soviet aircraft which fired warning shots had no tracer ammunition, so that those shots were invisible to the KAL 007 crew. But the primary cause was that the airplane was where it was not supposed to be.

In this particular case, I find it hard to accept Degani’s explanation: it seems incredible that during a long flight with low workload the crew did not attempt to verify their position by plotting the INS position manually, or by taking radio cross-bearings…which would not have been very accurate at the ranges involved, but would still have most likely shown that something was wrong. Still, in the appendix Degani gives other examples of airplanes that wound up way off course as a result of similar autopilot confusion.

One common thread in these incidents is that they could have been prevented, or at least limited in the seriousness of their outcomes, by cross-checking. If the Asiana pilots had been keeping an eye on the airspeed indicator, they would surely have realized the autothrottle problem in time to recover. The captain of the flight that broke the speed limit did observe an excessively high airspeed and take corrective action. If the anesthesiologists in the medical example had noticed that the blood pressure readings were absolutely constant, they would have caught the problem much earlier.   If the officers of the Royal Majesty had cross-checked their GPS position with the LORAN position, OR periodically set the radar to a longer range setting, they could have avoided the grounding.

But when people use an automated system on a regular basis, and find it reliable, it is easy for them to assume that it can always be counted on, and for the cross-checking to be omitted. Indeed, in some instances cross-checking many not be operationally feasible–is it really reasonable to expect that an anesthesiologist will notice that a blood pressure reading is NOT changing, if he has other duties that draw his attention away from the monitor?

To maximize the safety of human-machine integrated systems, it is essential (a) that the creators of this system consciously design the interface with to try and avoid “automation surprises,”  (b) that the humans who will operate the system understand its functioning in depth, including the interaction between separate components (like the GPS system and the radar/map plotter) and the behavior of those modes which are likely to be used only on rare occasions, and (c) that cross-checking, wherever possible, should be conducted religiously.

In a future post, I’ll talk about a fairly recent accident that involved not only the human-automation interface, but also the nature of decision-making in large organizations.