One of the life experiences that comes with being a three decade veteran of military procurement is you have been around long enough to know where all the important bodies are buried — case in point, the Boeing 737 MAX. What we are seeing in the two recent 737 MAX crashes is the the 20 year accumulation of professional toxic waste and decay in Boeing management that came with the first Clinton Administration’s cancellation of MIL-STD-499A SYSTEM ENGINEERING MANAGEMENT.
I was e-mailed a link today to this Daily Kos post on the 737 MAX :
Did Boeing ignore basic SW engineering principles?
Thursday March 21, 2019 · 8:34 AM CDT
and this passage just jumped out:
A few software engineering principles:
- Software engineering 101: validate your inputs.
- Software engineering 201: when something goes wrong, provide useful data to the human.
- Software engineering 301: for life-critical decisions, avoid single point of failure.
Until today, I had thought that aviation was *good* at software engineering. But my faith is shaken by the New York Times description today of what went wrong with the Boeing 737 MAX.
The above passes my professional “Bozo Test” of whether the poster knows what he is taking about regards software development. He does.
This is where that “military procurement life experience” I mentioned comes in. The timing of the development of the 737 MAX MCAS software was roughly 20 years after the Clinton Administration cancelled the majority of Mil-Specs in the mid-1990’s and in particular the one for system engineering management.
MIL-STD-499A (NOTICE 1), MILITARY STANDARD: SYSTEM ENGINEERING MANAGEMENT (27 FEB 1995) [NO S/S DOCUMENT]., MIL-STD-499A (USAF), dated 1 May 1974, is hereby canceled without replacement.
The 1992 Politics of Mil-Spec Cancellations
That Mil-Spec cancellation was related to both then Defense Secretary Cheney’s cancellation of the stealthy A-12 Avenger 2 attack jet and the subsequent support that then “Candidate for President Clinton” got from major American defense contractors.
This support of Bill Clinton was retaliation by the big US defense contractors for Defense Secretary Cheney holding large defense contractors accountable for a major defense program’s failure to perform. Killing that political precedent was a vital interest for American defense contractors in the 1992 presidential election.
The 1995 MIL-STD-499A notice of cancellation means the Boeing management involved with the 737 MAX software development grew up professionally after after DoD directed system engineering management ended…and it shows. Again from the Daily Kos:
- There are two “angle of attack” sensors on the plane, but the software that controls the position of the nose of the aircraft relied on just one of them.
- A previous Seattle Times article had explained that the software then fought with the pilots, refusing to put the nose where they wanted it.
- Boeing sells an optional(!) indicator to tell the pilot what the angle of attack sensor is sensing.
- Boeing sells an option(!) to tell the pilots when the two sensors disagree with each other, a “disagree light”.
- And in the biggest “no sh*t sherlock” that I have ever read about software:
In the software update that Boeing says is coming soon, MCAS will be modified to take readings from both sensors. If there is a meaningful disagreement between the readings, MCAS will be disabled.
Gasp. How could the plane have been released without the two options, and without this check for disagreement?
Systems engineering Mil-Specs were the institutional basis that the US Defense Department used for “inculcating” systems engineering rigor into (Read: Force down the resisting throats of) large government contractor corporate bureaucracies like Boeing.
MIL-STD-499A didn’t always work in defense projects, but the formal structure of this systems engineering Mil-Spec left little room for the Dilbertian “pointy haired” management types to play the games you saw in the 737 MAX.
It also gave good engineers room to do their thing because the preparation process formal Mil-Spec reviews distracted the hell out of the pointy haired management types from dreaming up things like making passenger flight safety into a profit center.
The rigor of the MIL-STD-499A SYSTEM ENGINEERING MANAGEMENT approach trained good contractor program and production managers. These managers were systematically stolen from the defense side of Boeing into the commercial side for difficult developments because they had the skills set to juggle/coordinate a whole lot of organizational and sub-contractor balls at the same time. This was what MIL-STD-499A required to meet its performance metrics. Point in fact, many civil projects of the era used MIL-STD-499A when systems reliability really mattered.
My 737 MAX Investigation Expectations
I don’t expect the main stream media to go anywhere near the information above both because of the “Narrative Implications” regards the Clinton Administration and for the very basic fact that the “credentialed but uneducated” 27 year old reporter with a journalism degree’s reporting on the 737 MAX has the following level of aviation technical skills:
I expect the MSM reporters will get used by everybody Boeing has screwed in the last 10 years and who wants pay back in exactly the same way that MSM reporters were used by both the Hillary for President Campaign and Comey era FBI with the “Trump-Russian Collaboration Conspiracy Theory.”
A Short Lesson in Systems & Reliability Engineering
This ignoring of the Mil-Spec cancellation connection to the 737 MAX is a shame. Since the basic practice of reliability engineering in hardware and systems is a very mature discipline. Basic mathematical reliability theory was solved during the 1950s. The best textbook every written on the subject is by Igor Bazovsky and was published in 1961. Bazovsky’s mathematical theories were codified in Mil-Hdbk-217, Mil-Std-756B and number of other systems engineering reliability standards. Bazovsky’s text book is both available as scanned PDF 1961 edition on multiple websites and as the 2004 hardcopy/Kindle edition ebook on Amazon here: https://www.amazon.com/Reliability-Theory-Practice-Mechanical-Engineering/dp/0486438678 .
Software reliability principles are also well understood, if lacking the strong theoretical foundation in systems and hardware modelling techniques that Bazovsky pioneered.
For an introductory understanding of how mathematical systems reliability models work with hardware and insights on the software/hardware interface, see the following systems engineering PDF document applying reliability formulas on the WW2 era P-38 Lightning fighters and several modern software/hardware systems failures:
System Reliability and Metrics of Reliability
Peter Harding & Associates, Pty Ltd
Copyright 1996, PHA Pty Ltd, All rights reserved
Summary & Closing:
These system reliability mathematics that apply to 737 MAX MCAS flight software & sensor hardware development are accessible to anybody with a STEM university education…but the life experience that tells me about where this particular body is buried also tells me mainstream media reporters will leave it buried.