Another Software Debacle

The project to computerize immigration documentation is not going well…after $1 billion in expenditures and and a new estimated total cost of $3.5 billion—the original estimate having been for half a billion.  (via Marginal Revolution)

And, of course, we’re all aware of the problems with the Obamacare website and supporting back-end systems.

Back in 2006, I wrote about the failure of the FAA/IBM project to develop the “Advanced Automation System” for air traffic control: The story of a software failure, based on the writing of Robert Britcher, who was involved in the project.

The problems with the ATC project were to my mind somewhat more excusable than the ones with the current immigration project:  the “Advanced Automation System” was required to operate with extreme reliability and availability with stringent real-time response criteria, to interface with radar systems, and to support a complex and safety-critical user interface.  The immigration system sounds like basically a large database and workflow system.

18 thoughts on “Another Software Debacle”

  1. Perhaps they don’t actually want immigration documented. How do Democrats benefit from that?

    And on the subject of air traffic control, Boeing claims has developed time and fuel saving systems:

    Tailored Arrivals:
    A Smart Approach to Landings:

    So as usual, while the government dithers and tries to maximize political advantage, the private market innovates and solves their customers problems.

  2. The FBI spent even more and finally gave up. I posted on that before. Air Traffic Control has a solution. Make it private.

    Canada has and it is successful.

    NAV CANADA is the country’s private sector civil air navigation services provider. With operations from coast to coast to coast, NAV CANADA provides air traffic control, flight information, weather briefings, aeronautical information services, airport advisory services and electronic aids to navigation.

  3. I think privatization of ATC is highly questionable. My view is that privatization makes sense in those cases where there is a market test of the providers, and this inherently does not exist for ATC. You cannot have two competing organizations managing the same airspace and giving users a choice as to which one they want to deal with. You could split the country into 5 pieces and give each one to a different contractor, but this ignores the question of common procedures and common systems.

    Most likely, privatization of the US Air Traffic Control System would result in the contract being assigned largely on the basis of political considerations.

    Is there really any evidence the Nav Canada works any better than US ATC, taking into account traffic volumes and mix and airspace complexity?

  4. Another solution might be to create a specification of performance goals, then let out bids for a system. Allocate some money for software & hardware development and system integration for a demonstration and evaluation phase. Have a government/industry team vote on a winning system or vote to try again.

  5. There are 22 Air Traffic Control centers in the US. There is no reason why they could not be put up for bid individually. We have more than one phone company but a national phone network.

  6. The ATC centers (with their surrounding towers and TRACON facilities) could be put up for bid individually, but there would need to be common software, network standards, and operational procedure definitions…which is the hardest part. The Centers would be conducting operations…how would you measure which ones were most worthy of having their contracts renewed? How much should a near miss count versus a traffic delay? What would be the incentives to work smoothly with the adjoining regions?

    I don’t think the phone analogy is a good one here. If I use Comcast (shudder) and my neighbor uses Verizon, there is no conflict. A closer analogy would be if exclusive franchises were handed out to various communication carriers in each of 22 regions of the country.

  7. “there would need to be common software, network standards, and operational procedure definitions…which is the hardest part.”

    There isn’t now. The last I checked they were using 1970s systems with marginal upgrades.

  8. The system which is in use at most radar approach facilities (TRACONs) is the Common ARTS system. It was originally implemented on Univac systems in the late 1960s; the Wikipedia article says that in the 1980s the software was rewritten in C for a microprocessor environment, and has later been ported to a PowerPC environment by Lockheed Martin. There have been considerable functional improvements over time, including the interfaces required to support ADS-B data (in which aircraft report their own GPS-derivered position data as an alternative to radar location)

    Common ARTS is supposed to be replaced by STARS:

    …which is indeed having difficulties:

    My point is that there is indeed common software which is networked together and which handles thousands of flights every day, mostly without incident. Midair collisions between aircraft under ATC control are very, very rare.

  9. IT contracting is where the modest frauds go to grind out a living. Big Fraud is Wall Street, see HFT.

    No, it’s not all fraud. But it’s the accounting scam [remember Arthur Anderson driving the getaway car] of our time.

    If you’re raking it in you’re not incompetent.

  10. Seriously, does anyone actually expect the government to ever do anything particularly well?

    Even progressives acknowledge this when the subject is the military, or the intelligence groups, or law enforcement. Then, they will point out all the failures, spending problems, and general incompetence of the practitioners of things they don’t like.

    Somehow, in a form of self-delusion I have never understood, they will not examine all their favorite programs with the same relish, nor admit that all the human service/welfare programs are every bit as uneconomical and incompetent.

    I understand why they do it, I just can’t perform the mental gymnastics to see how they manage it.

    Of course, it’s an obvious argument for privatizing a whole bunch of this stuff, or just disbanding the government agency and letting things develop without interference from a bunch of pols, but that idea is so terrifying to progs that it will never occur to them, no matter how bad things get.

    When you’re a one-trick pony, the only answer is more, more, more.

  11. “Seriously, does anyone actually expect the government to ever do anything particularly well?” Megan McArdle argued some time ago that one reason so many Americans despise and distrust the Federal government is that it is unusually incompetent. Compared, I suppose, to national governments in other parts of the advanced world, and perhaps compared to the better US state governments too?

    Hm: I see I originally mistyped that as “Feral government”, a happy error.

  12. Theodore Dalrymple (Anthony Daniels) has an excellent column on the negative aspects of an efficient government.

    It’s called “The Uses of Corruption.”

    Some of us care about what the intentions of government are before we worry about its efficiency. Mussolini made the railroads run on time, for example.

  13. The phone company comparison is apt in my mind because I can direct dial from a Bell number to a CenturyLink number without ever being aware that I have crossed a border. I am sure there is some protocol that would have to be designed to facilitate transfer of a flight from one ATC to another. But that doesn’t seem insurmountable. But there does not seem to be any reason way all ATC’s must run the same software within the ATC as long as they can pass flights to another. Having two sets of vendors to operate ATC’s would stimulate improvement and reward success.

  14. “some protocol that would have to be designed to facilitate transfer of a flight from one ATC to another.”

    The programmers of 50 years ago knew how to do this. They called them subroutines.

  15. Back in the 80’s, I was in charge of a computer installation at my work place. The previous attempt had failed, and the after-action critique that several of us did indicated that the major problem was the number of sub programs that had been attached to the main task program to generate administrative data.

    Our response to any requests for similar sub programs during the new installation was very simple—you can have all the admin data you want tomorrow, but nothing runs during real time operations except that which is required to accomplish the primary task.

    The installation was successful, and it ran for several years before a major update was needed. My friend was the technical director for the overhaul, and we spoke several times about the potential problems and issues that might arise. Once again, his primary rule to all requests for extras was that any admin data could be extracted after 24 hours, but he would not allow cumbersome add-ons to be run in real time.

    The update was successful, on time, and within budget.

    The reason so many of these governmental programs flounder is not lack of analytical skills or programming expertise. It’s because powerful administrative people within the agency, and from other agencies, barge into the design and implementation phases demanding their pet stuff be included, with little regard for the ultimate effects all the extra stuff might have on how the system runs.

    The situation is similar, in many ways, to the problems the major automakers were having several decades ago, when they were the “top dogs” in the auto world, with getting a new car design approved and made. I remember an article from that period, probably the 1970’s, describing the tortured route through the layers of administrators and non-engineering people any designs had to navigate before anything could actually be manufactured, even in prototype.

    The result, of course, were cars which were poorly designed, poorly made, and lousy to own and drive. In the final analysis, all these problems were management failures, and they nearly destroyed the US auto industry.

    One of the major reasons statism fails repeatedly regardless of the problem it is attempting to resolve is the utter politicization of the process, and the endless ranks of time serving, empire building, interfering apparatchiks who waste so much time and resources, and endlessly complicate any attempt to put a system together no matter what object of it is, that the sheer weight of all the kludges just sinks the desired results beneath a sea of nonsense and political garbage.

    I was taught that one way to judge any system is by evaluating its product. Given the malfunctioning, and often counter-productive, efforts of our ruling elites, opposition to their controlling any more of our lives, or continuing to run the areas they’ve already taken over, seems founded in very realistic and practical considerations.

  16. Protocols for the transfer of flight data between ATC regions certainly exist: they are communication message formats, and they are no doubt invoked by subroutines.

    Someone has had to establish what these formats are, under what conditions they are to be transmitted, what the receiving entity (both the software and the people) is to do when they receive them, etc. Similarly, protocols have had to be established for interchange of data from radar systems and (more recently) from ADS-B position-self-announce systems.

    It would be possible for multiple vendors to write interoperating ATC systems…indeed, this interoperation already takes place between the Regional Center systems and the Approach Control (TRACON) systems, also between different generations of the same function…viz the ARTS and STARS systems…but there needs to be an overall system architecture to which they all adhere if they are to properly interoperate. It is also highly desirable that there be consistency of user interfaces if controllers are to have job mobility across regions.

  17. My rule of thumb when dealing with people is simple. It should work pretty much universally.

    To wit:
    Reward the behavior you like.
    Punish or starve(of funds) behavior you don’t like.

    Duplication of effort in writing code to control flight information seems wasteful. In addition to the initial duplicative effort, you have to consider the on-going maintenance effort to keep the software functional when ‘things’ change. And they will.
    The initial double cost of development will shrink in time compared to the cost of maintenance. Eventually, you will find you do not have live people readily available to keep things working. As we have to a degree right now.(last I knew)
    Let the government ‘experts’ propose a basic spec, and let the airlines review it. Compare to off the shelf systems from current sources, and get a RFP prepared and out the door. Go on vacation. Read the proposals, and select your choice. Take another vacation. Blame the innocent, reward the guilty, and take a promotion to another job days before implementation. BTW, don’t take a plane on those vacations or that exit….


Comments are closed.