Since the 1980s, automating various flight management operations has contributed to a profound improvement in air transport safety and effectiveness. But a related human issue — automation dependency — has emerged as a significant challenge to further improvements in safety levels. Automation can contribute to diminishing manual flying skills and increasing complacency, as pilots avail themselves of automatic flight management and navigation systems to aid much of their decision-making.
In some cases, pilots don’t fully understand the automatic processes controlling their sophisticated aircraft. The ironic enquiry “What’s it doing now?” is sometimes heard in the cockpit, as pilots struggle to figure out the actions of the “automatics”, as these systems are referred to on the flight deck.
Crucial as this is in the cockpit, automation dependency is equally problematic in many businesses today, whenever there is a disconnect between what managers think is going on and what is actually happening. The automation in question is not just technological, but also pervades the processes, algorithms, and reporting on which managers rely to inform their decision-making.
Aviation is addressing this phenomenon as a major problem and is seeking solutions. We suggest that many companies should do the same before their “business automatics” put them at risk of losing control.
RTWT
Reminds me of this aviation classic, a 1997 talk by an American Airlines pilot: The Children of Magenta.
See also: Automation is Fragile, People are ‘Antifragile’ and my posts When Humans and Robots Communicate and Blood on the Tracks
LTCM
LTCM = Long-Term Capital Management….did I get it right?
Aren’t several of the Wall Street ‘Black’ days since about 1985 blamrd on automated trading programs?
Yes, Long Term Capital Management. Their systems worked well until the mkts had the equivalent of a 500-year flood, which happens in mkts every ten years or so.
I don’t think this kind of financial-mkt systems problem has a solution.
Not just business or airplanes…..
My wife and I took her visiting cousin on a tour boat trip here in SW Florida a few weeks ago. As the boat was leaving the dock of an island restaurant after the lunch stop, we were involved in a collision with a very pricey luxury runabout. The guy hit the tour boat amidships as he approached the dock at about 5 knots, without slowing down. I was shocked, as I had been watching him approach, waiting for him to slow and reverse to stop….he had plenty of time to do so. He kept looking at the tour boat, then down at his dash, then at the tour boat, then down, several times until he crashed into us. His explanation turned out to be that he was using his auto-pilot and it had “just quit working”! It seemed obvious to me that he didn’t know (or couldn’t remember) how to get manual control back.
Manufacturing consultant Bill Waddell, about managers who know the numbers but don’t know the product
Bill also passes along a good observation about the connection between metrics obsession and lousy organizational culture.
“he didn’t know (or couldn’t remember) how to get manual control back”
That’s the summary of the 20th century right there.
The problem isn’t automation per se, it’s overconfidence and arrogance which are part of human nature. It’s a failure of education, because the systems users in these cases did not understand the basics and were too trusting of the automation. That’s why managers should understand the product and have direct experience with manufacturing and marketing, pilots should know how to fly the plane without autopilots or computerized navigation systems, and hedge fund guys should be more cautious about placing big bets based on past market activity.
With the current vogue for ‘Big Data’, I’m confident that we will see several debacles caused by placing too much confidence in impressive-looking sets of numbers, without really understanding with they mean.
I work in the Electric power generation business and we see the same thing. I was in a power plant where they had used an old control building for new power unit. Along the walls of the old controlroom were hundreds of dial gauges and displays. I asked one of the older operators if he though the new computer display systems were better. One thing he noted was that they visually knew the expected behavior of the dials and could scan them all rapidly. The computer systems required you to dive deep into the various sub windows to see a lot of information. So as an overview, the old system with a good operator provided much better situational awareness.
In this case even if a control system was not performing you would see rapidly vibrating needles, needles pegged high or low etc. This would allow you to take action quickly
U. S. Navy has recognized this could be a problem.
My wife and I were reading the VF story about the audio tapes recovered from the sunken ship El Faro.
That reminded me of the story about the Air France flight 447. The Vanity Fair article, did a good job of explaining what happened.
Here is an example of what I used to call the “DC 10 cockpit syndrome” in surgery. Too many decisions and not enough time.
Th example I was referring to was the crash of a DC 10 on takeoff in 1979 when an engine came off the wing during takeoff.
Somebody on Facebook posted a photo of a group of old time Pan Am airline captains. They flew all over the world but, when airlines got modern, they got rid of the old captains because they were too “Independent.”
Automation required less “independent” pilots. And so we got KAL flight 007.
The H/E scenario additionally suggests the flight’s first officer did know they were flying away from the planned course, but the airline’s culture discouraged anyone from questioning the captain’s conduct of the flight, so he remained silent.[12]
The theory that the INS system was set incorrectly gains considerable credibility if the following are considered:
Although there are three independent inertial platforms, when in the ramp position, the operator only inputs one initial position in order for the platform to “erect”.
This would have been done by the Flight Engineer alone, before the other crew members arrive.
In order to erect properly (that is, enter gyro-compassing mode), each platform relies on the correct latitude, but not the longitude.
Therefore, if the longitude was incorrectly set, all three platforms would seem to erect normally, but with a position some 10 degrees in error.
Wikipedia indulges a few left wing conspiracy theories but most people are satisfied that navigation error was the cause.
The best commercial pilots have been the former military pilots. But that sup[ply is dryinkg up. 2 crashes that i can think of – Air France 447 and the Buffalo commuter crash – both seemed to involve pilots really not aware of basic airman ship and relying on flight directors and computers.
Heck, throw in the Korean 777 that crashed at SFO. Too low and too slow. I am sure there others.
A basic understanding in any endeavor is the foundation.
Mike….the link for the Surgery post is missing
“The computer systems required you to dive deep into the various sub windows to see a lot of information.”
So Musk is right, AI is going to kill us, or allow those that can harness it to do so, at will.
Death6
There’s a big difference between UI/UX problems and automation problems. Let’s not conflate them.
The main threat from AI is to people who shut down their own brains and do whatever they are told, whether it’s driving off a bridge because Google Maps told them to, or thinking that Tesla Autopilot is actually capable of driving a car itself. Let’s not buy into the hype of these silly techno-futurists, who also believe nonsense like The Singularity, etc.
DirtyJobsGuy has a good point. Human pattern matching is a skill we often ignore in our design of control systems.
The last nuclear power plant design I worked on had a fully automated system to do startups – including pulling control rods. I thought that was going too far – the humans should make the decision as to when the reactor is critical.
As to alarms, we used to have too many annunciator windows but now they are ganged into “smart annunciators” that only give immediate high level alarms. You do have to dig down through layers of screens to find out the root cause.
Sometimes simpler is better.
“Mike….the link for the Surgery post is missing”
The syndrome is just my own invention to explain to young surgeons how things can get out of control. I discuss it in my second book.
The link to the article about the actual crash is there, But I see the link went bad. Here it is.
The National Transportation Safety Board (NTSB) found that as the jet was beginning its takeoff rotation, engine number one (the left hand engine) separated from the left wing, flipping over the top of the wing and landing on the runway. As the engine separated from the aircraft, it severed hydraulic fluid lines that locked the wing’s leading edge slats in place and damaged a three-foot section of the left wing’s leading edge. Aerodynamic forces acting on the wing resulted in an uncommanded retraction of the outboard slats. As the jet began to climb, the damaged left wing, with no engine, produced far less lift (stalled) than the right wing, with its slats still deployed and its engine running at full takeoff speed. The disrupted and unbalanced aerodynamics of the aircraft caused it to roll abruptly to the left until it was partially inverted, reaching a bank angle of 112 degrees, before crashing in an open field by a trailer park near the end of the runway.
The engine disengagement causes a cascade of failures.
That scenario was added to the simulator after that crash.