My profession is much in the news at the moment, so I thought I would pass along such insights as I have from my career, mostly from a multibillion-dollar debacle which I and several thousand others worked on for a few years around the turn of the millennium. I will not name my employer, not that anyone with a passing familiarity with me doesn’t know who it is; nor will I name the project, although knowing the employer and the general timeframe will give you that pretty quickly too.
We spent, I believe, $4 billion, and garnered a total of 4,000 customers over the lifetime of the product, which was not aimed at large organizations which would be likely to spend millions on it, but at consumers and small businesses which would spend thousands on it, and that amount spread out over a period of several years. From an economic transparency standpoint, therefore, it would have been better to select 4,000 people at random around the country and cut them checks for $1 million apiece. Also much faster. But that wouldn’t have kept me and lots of others employed, learning whatever it is we learn from a colossally failed project.
So, a few things to keep in mind about a certain spectacularly problematic and topical IT effort:
- Large numbers of reasonably bright and very hard-working people, who have up until that point been creating significant wealth, can unite in a complete flop. Past performance is no guarantee, and all that. Because even reasonably bright, hard-working people can suffer from failures of imagination, tendencies to wishful thinking, and cultural failure in general.
- Morale has got to be rock-bottom for anybody with any degree of self-awareness working on this thing. My relevant moment was around the end of ’99 when it was announced, with great fanfare, at a large (200+ in attendance) meeting to review progress and next steps, that we had gotten a single order through the system. It had taken various people eight hours to finish the order. As of that date, we were projecting that we would be doing 1,600 orders a day in eight months. To get an idea of our actual peak rate, note the abovementioned cumulative figure of 4,000 over the multi-year lifespan of the project.
- Root cause analysis is all very well, but there are probably at least three or four fundamental problems, any one of which would have crippled the effort. As you may infer from the previous bullet point, back-office systems was one of them on that project. Others which were equally problematic included exposure to the software upgrade schedule of an irreplaceable vendor who was not at all beholden to us to produce anything by any particular date, and physical access to certain of our competitors’ facilities, which they were legally required to allow us into exactly two (2) days per year. See also “cultural failure,” above; most of us were residing and working in what is one of the most livable cities in the world in many ways, but Silicon Valley it ain’t.
- Not to overlook the obvious, there is a significant danger that the well-advertised difficulties of the website in question will become a smokescreen for the fundamental contradictions of the legislation itself. The overall program cannot work unless large numbers of people act in a counter-incentived (possibly not a word, but I’m groping for something analogous to “counterintuitive”) fashion which might politely be termed “selfless” and do so in the near future. What we seem likely to hear, however, is that it would have worked if only certain IT architectural decisions had been better made.
This thing would be a case study for the next couple of decades if it weren’t going to be overshadowed by physically calamitous events, which I frankly expect. In another decade, Gen-X managers and Millennial line workers, inspired by Boomers, all of them much better at things than they are now, “will be in a position to guide the nation, and perhaps the world, across several painful thresholds,” to quote a relevant passage from Strauss and Howe
My only complaint is that it is not available in Kindle/
if it weren’t going to be overshadowed by physically calamitous events
Care to expand on that topic?
So, cutting a very, very long story into digestible sections……….
…….You and the others wouldn’t accept you-all got it wrong, and blew it!
One of the most famous stories of a large-scale software failure was the FAA/IBM project known as the Advanced Automation System. I wrote about it here:
http://photonplaza.blogspot.com/2006_03_26_photonplaza_archive.html
This project was probably technically more challenging than the Obamacare project, given that it involved real-time response demands and extremely high availability requirements; OTOH, the Obamacare project was plagued with hundreds or thousands of requirments that were set in legal/regulatory stone (to the extent that they could be disambiguated at all) and a finish date that was not allowed to move.
The usual problem with development failure is generally stupidity, wishfulness, and incompetence on the highest levels of the project.
People laugh at the pointy-haired boss of Dilbert without grasping that’s sometimes them. The Peter Principle applies to all, not just people with pointy hair.
My own experience with failed software projects include:
1) A company that failed because the Man-in-Charge failed to consider marketing expenses when setting up his business plan. Yes, he figured if he built it they would come.
2) A company that failed because it tried to write a system for a decrepit box (an old mainframe) when its needs could have been better fulfilled by available microcomputers of the time. It also spent money on a suite of offices long before it had a product, and essentially went through about 200k in 1985 dollars before abandoning the mainframe for a microcomputer design, then ran out of money before it could afford to finish the development.
3) A successful company that owned its market but then failed because the salesmen got in charge of it and promised anything to make a sale, which quickly perverted the software into a mass of unsupportable kludges.
4) An established architectural company that opened a new 15-employee satellite office in a major city at a cost of about half a million. Never secured a single significant new client before they closed its doors. Another “build it and they will come” effort. This was a company with a sales/marketing staff, mind you, and more than 30 years of business operation experience in at least one employee.
5) A software testing project with finish dates created by a documentation specialist bumped to PM because that was the only way to go upward for him, despite zero educational background in software development, just some self-taught coding experience. Note that this was not an Agile house… His development plan was laughable, allocating, for example, 30 days from software handover to finalizing the whole regression test process for a large software module. That’s exploratory, test plan, and coding. Final *testing* code was like 3000-9000 lines of pure testing code (3000=minimal regression tests implemented, 9000=thorough regression tests implemented). Yeah, piece o’ cake. :-/
}}} Large numbers of reasonably bright and very hard-working people, who have up until that point been creating significant wealth, can unite in a complete flop. Past performance is no guarantee, and all that. Because even reasonably bright, hard-working people can suffer from failures of imagination, tendencies to wishful thinking, and cultural failure in general.
I do not accept the notion that a common, much less typical cause of failures are because of anyone operating at lower than the top two tiers of the decision making process, and usually it’s at the top tier.
Disasters come from the top down. One thing I learned just from watching those early number of small business failures was simple: Spend cash flow however you wish, but spend reserve capital like it was inches off your dick. Most of what I have heard has agreed with this principle, as well. This was the primary failure of 4, and I suspected it at the time, but presumed the “boss” knew what he was doing. But he was the boss, and there’s no way I’d have questioned his judgment without being asked. Turns out I was right, and he did not know what he was doing. He should have opened a very small office to get the business initiated then expanded as cashflow built up. Eventually he got fired and the company downsized almost into non-existence from 55 employees at peak.
I’m not saying that the lower level employees can’t screw up — I’m saying that the only people who make DISASTERS are the top level ones making bad decisions about key elements, key employees, and key policies.
That’s the problem with hierarchical business design, the information flow is not effective, since it’s all channeled through the upper layers, and only those upper layers can really make decisions based on the information. The larger the hierarchy, the less likely the decisions will be effective, much less optimal, and sooner or later one or more of those are going to bite you in the ass and take out one of your cheeks.
The need here is for a more networked, “unit” sized management system, where failures are more likely to be strongly localized, and information flow is less limited. Who knows, maybe if I’d been more encouraged by the system to ask the questions in my mind @4, someone might have thought about it a bit sooner, and not gone so whole-hog with the new office. Not saying my own voice should have been listened to alone — but others with experience might have said, “yeah, you know, that’s right…” and a consensus of opposition might have made the idea less palatable to him. But it was very much a hierarchy — I was encouraged to recommend in my own arena of expertise but not on general thoughts and ideas gained from experience.
My own (then) small company had its own experience in a software project failure. A competitor was offering something to our biggest customer, a well-known oil company.
Near panic set in – this oil company had been our critical customer for years and we had nothing to counter.
So I start to work – and hire a friend, who was/is also – I would say – one of the top 1-2% of programmers. I could tell you stories but rather than derail my tale will say but for a few changes he could have been another Steve Wozniak.
Anyway we start to get to work and my friend’s suggestion was to get the thing to market ASAP, known bugs and lack of desired features notwithstanding.
I would call this the Microsoft model, cynicism aside).
But I was the perfectionist, wanting everything working right – and I even worked as a service writer in a service station for a couple of months to understand what the process was and adapt my software to their needs. (which as yet another aside I note is a huge deficiency in so much software today – from POS (that is point of sale not the other although it could be used) – anyway sliding your ATM card to auto GPS devices to….
Anyway long story shorter I am finally ready for market just as Windows is taking hold.
I had the best DOS-based program …..
It only cost me about $120,000, not $4 billion ;-) But my labor was cheap ;-)