Complexity Choking

I was literally in the middle of writing a post about Microsoft’s (MS) problems shipping Vista when I took a break, clicked over to Slashdot and found a link to NY Times article that covers most of what I intended to say.

Well @#$%&!

So, read the article and then, if you want, you can read my thoughts as well.

Many people look at large organizations like Microsoft, Walmart or governments and see only the advantages of size. Few stop to consider the disadvantages. Economists have long recognized a counterpoint to economies of scale called diseconomies of scale. As organizations get bigger, their complexity often rises non-linearly. If a firm doubles in size, its complexity (the amount of information it must process) may not double but triple. It takes time to process information, and as the information burden grows the decision cycle for the organization slows. The organization can no longer respond to changes quickly. Many companies have literally grown themselves to death. Lets call this phenomenon complexity choking.

Microsoft is particularly vulnerable to complexity choking because it bases its entire strategy on tying its many, many products together so that customers are encouraged, if not forced, to chose MS products for virtually all their needs. However, that means that each new product must be fitted into the existing matrix of products. New versions of the operating system must support applications written for older versions. Applications can’t be simple single-purpose tools but must interoperate with other applications. This is not even considering what happens when the guys in marketing start writing specs.

Microsoft’s mania for interdependence and interoperability is also the major cause of their security problems. They can’t apply modern concepts of security without breaking older software, and the interlinkages between applications allow malware to spread quickly.

Microsoft’s broad market share also works against them. Each Microsoft product must meet the differing needs of each segment of the market. Often these needs do not overlap. For example, a study once showed that 80% of the users of MS Word never used more than 20% of the features provided. However, those 80% of users divided into many subgroups, each using a different subset of the features. MS Word is such a monster because in order to attract enough users to become a de facto standard it must include a large number of features that only a small minority of users will ever use. The operating system faces the same hurdle. It has to meet the needs of everyone from emailing grandmas to corporations running databases.

I have real doubts whether Microsoft’s business model can even survive for more than a decade. The more they try to tie everything together the more they choke on complexity. Customers too have to deal with complexity overhead. Eventually, the cost of doing so will outweigh the advantages of interoperability. When that point is reached, Microsoft is dead.

13 thoughts on “Complexity Choking”

  1. Don’t worry Shannon, you said it better than they did.

    —-

    I liked what you said about “Microsoft’s mania for interdependence and interoperability”

    Up until a few years ago, the desktop/laptop was the whole system. Now the desktop/laptop is just another component in the system. Desktops, laptops, Ipods, PDAs, TV’s, home stereos, car stereos, camera’s, phones, home security. In the next few years they’ll wirelessly stream everything but AC power.

    Apple has worked hard on the “interoperability” part, making components for the system and getting them to work together out of the box. Meanwhile, as you pointed out, MSFT is still stuck on “interdependence”, trying to tether all our components with it’s uber-OS.

    All this has generated alot of ill-will. Outside of big-oil, is any company viewed more negatively than msft? As soon as there is an alternative, that ill-will could really come back to hurt them.

  2. I have owned and operated microcomputers since 1979. From the point of view of the operating system, the internet, video streams, audio streams, cameras, scanners, etc are all peripherals. It is the job of the operating system to manage these resources seamlessly as well as logically. If Microsoft failed to adapt the operating system to work with external peripherals (to the cpu box), then they have failed to meet their core mission and would have deservedly died years ago.

    All useful systems are inherently exploitable. Just as the biological world has parasites, the electronic world has them too. Microsoft’s business model has been and remains adaptable.

    Is Linux more secure than Windows? Probably not, since it has not been stress tested to the extent Windows has. For me, Windows is worth my money. Windows is more valuable to me than writing my own operating system, using Apple systems (which I have in the past but no longer precisely because Apple equipment did not interoperate well with anything but its own peripherals), using Linux (didn’t work with my equipment), or doing without. Microsoft works well with peripheral makers so that there are drivers for nearly every kind of peripheral you can think of.

    Interdependencies? Only in some folk’s minds. Microsoft’s API’s (the things needed by programmers to use operating system services) are generally public and reasonably well documented (there are 1000’s of them and the documentation spans several gigabytes of data).

    Unix was no more secure at its birth than Windows was; and being networked from nearly the start, it probably wasn’t as secure. The first net worm attacked Unix, after all, simply because it was there as a networked system.

    The internet was not designed to be secure either. The internet was designed to ensure message delivery so long as an available path exists. The internet was also not designed to be a publicly available system. The intenet became the wide open communication system it is today precisely because it is robust and useful; which is why it is so readily exploited for malicious use today.

    Shannon, tying things together is the thing that an operating system must do in order to be useful. Program–operating system–local peripherals–external world. Back in the bad old days, you had to do this tying together for yourself if you wanted to get anything useful from the system. Now you don’t! If Microsoft had not done things the way they have, then they would have deservedly died and *WE WOULD BE POORER FOR IT*.

    I wish folks would quit mischaracterizing Microsoft (they are not a monopoly…no one makes you use them or you die), quit whining about their successes, and gloating over their failures. Like government, Microsoft is not G-d nor good; but Microsoft is useful, and necessary as long as Microsoft fulfills its mission.

  3. Charles, I don’t think one has to dislike Microsoft to see the validity of Shannon’s point. Companies often *do* tend to tie themselves in knots by overemphasizing the connections among their products and/or services. This reduces the ability of division managers or product line executives (whoever is responsible for individual product lines) to make decisions in an autonomous and timely fashion, and forces too many of these decisions to either be made at the top corporate level, or to become bogged down in “consultative” committee processes.

    It’s always easy to make a case for the linkage between product lines in particular cases; too often, the downside isn’t considered until it’s too late.

  4. I don’t think Microsoft is quite as doomed as Shannon thinks it is. Interoperability between apps is not as important as interoperability between users. What will keep Microsoft alive is network effect.

  5. Charles D. Quarles,

    I actually intended my observations to be something of a counterpoint to the idea that only state intervention can constrain large corporations. The diseconomies of scale will often do the job naturally.

    Microsoft has made design decisions based on non-technical marketing reasons that have created problems with security. The interaction of IE into the file browser, the auto-executions of mail scripts and other macros. In other cases, the need to maintain backwards capability prevents them from enforcing a real permissions system.

    I think in some cases however, bad decisions on Microsoft’s part don’t have much to do with it. For example, the very fact that Windows runs on the majority of systems means that it will always be the primary target of malware writers. That’s not Microsoft’s fault but it is a reality for their customers.

    I do think that Microsoft is trying to do to much. Right now they are trying to create an operating system that will run everything from the living room “media center” to corporate servers. That is to much range. I think they will find that in trying to do everything they will do nothing well.

  6. Steven Den Beste,

    If my thesis is correct, the network effect is actually part of the problem. The attempt to spread the network wider and wider makes the overall evolution of the system progressively slower. At some point, the loss of progress will overwhelm the advantages of using what other people use.

    Microsoft might be in trouble now because they sold a lot of assurance contracts based on the premise that Vista would ship this year. The contracts expire at the end of 06.

  7. “80% of the users of MS Word never used more than 20% of the features provided.”

    I doubt that it is 20%. I don’t think I get close to 20%.

  8. Shannon,

    I am not paid by Microsoft, but as a long term willing user and beta tester, I dispute your contention that “non-technical marketing reasons” affect design decisions. The crucial technical design decisions to reuse old code to the greatest extent possible and to maintain backward compatibility to the greatest extent possible certainly have had a down side security wise. Yet people forget that Microsoft has had to move from a single user, single tasking system to a multi-user, multitaking, internet connected system. Microsoft is one of the few companies that I deal with that goes out of its way to solicit outside user and developer feedback about its design decisions. Lord knows I wish I could work for them, and I think Steven Den Beste has nailed it! The Vista delay will not affect the big boys, and I am sure that Microsoft will work something out for this winter’s personal use buyers with folks like Dell.

    Your points about IE’s use as a dll that provides screen rendering services for Explorer are illogical to me. As a programmer, macro programmability enhances the use value for me; and that is the technical reason for its inclusion (developers pointedly asked for it).

    I run XP as a limited user, and I have no problem with its ACL permissions (and these are key to the file system security model). They are easy enough to work with and they are a real permissions system. That people misuse them is not Microsoft’s problem, though Microsoft does its best to try to prevent problems; many of whom cannot be known until millions of users are given time to bring them to the fore.

    User education will have to suffice. Nevertheless, any useful system is inherently exploitable. I don’t care what certain zealots say. The characteristics of a system that make it useful also make it exploitable. Too many folks refuse to see TANSTAAFL in any real world system.

    Ahannon, I respectfully disagree with you about the range of an operating system. Every OS vendor that I know of does the same thing Microsoft does simply because they must if they want to remain in business. An operating system manages resources for the running programs using the system. Calling the system a server or a media center simply does not logically change what the operating system services have to do.

    David,

    While I understand your response, I do not accept Shannon’s premises as valid; particularly given my own history with Microsoft products. Yes, that is anecdotal in nature; but the truth is the truth.

    Robert,

    What you say is rough rule of thumb; but when the day comes that requires use of one of those 80%, you’ll be glad you had it instead of wishing you had it so that you could get your work done and have to search the world over to find a product that does have it.

  9. Charles D. Quarles,

    I think it is naive of you not to consider that marketing has driven design decisions at Microsoft. The inclusion of IE into the file browser was clearly intended to guarantee that IE could not be separated from the OS in order to make it the web browser standard. The decision to do so was criticized at the time by every software security expert in the world (no exaggeration).

    Windows has the worst security of all major operating system for many different reasons. It is nearly impossible to run a contemporary Windows machine online for more than a few days before it is hopelessly contaminated. Other OS’s do not have that problem.

    You keep arguing like I am gunning for Microsoft. I am not. I am merely pointing out the inherent tradeoffs in trying to make a product that does everything.

  10. Microsoft and Intel are in cahoots.

    Neither of them would be near as profitable without the other. Intel designs faster and more complex CPUs. Microsoft designs slower and more bloated OSs.

    Without the faster CPUs Microsoft couldn’t sell its bloated code. Without bloated code Intel couldn’t sell nealy as many CPUs.

    I say this as a computer user since April ’75 (when you had to roll your own) and as a veteran of the clone wars – Zenith Data Systems. The rule of thumb at Zenith was that if code ran for 20 minutes or more without a problem – ship it. “Why” I asked? I was told – “what do you think the reset button is for. BTW for stress testing we used Doom.

  11. Shannon,

    I still reject your premise, and it isn’t from naivete. It is from a very long history of use and testing of microcomputer systems. I also know that it isn’t a “marketing” decision to supply the things your customers are demanding, nor is it a “marketing” decision to second guess your customers.

    My machine is connected 24/7 behind a NAT router and is fully patched. It is not hopelessly contaminated because I see to it that it does not get contaminated. I also know that every OS is exploitable, and that the others simply have not been attacked nearly as much as Windows simply because it isn’t worth the relative effort. I also know that using IE as a dll for screen rendering for Explorer makes perfect technical sense (as a developer…no need to write two renderers or two html renderers for screen output). I have four web browsers on my machine. Two of them can use IE as a dll for screen rendering.

    To test my thoughts, I ran depends.exe on the explorer.exe process and I found three IE related dlls, ieframe.dll, iertutil.dll, and ieui.dll. Any browser program, file system or web, running under any operating system would still need operating system services to function. If the OS directly provides these, why rewrite them?

    Logically, to the operating system, the internet is just another peripheral. Therefore, an operating system that did not know how to deal with the internet as a peripheral would not be able to compete with one that does.

    Again, the internet was not designed to be secure from attack against its hosts. The internet was designed to ensure message (packets) delivery as long as a path to the recipient exists.

    Where I emphatically disagree with you, Shannon, is this statement: “I am merely pointing out the inherent tradeoffs in trying to make a product that does everything.” The only “does everything” that the operating system must do, logically, is treat every potential peripheral, whether locally connected or not as locally connected input and/or output for the programs that run on the system. And that is just what Windows must do to survive as an operating system. The day that Windows ceases to do that is the day I look for someone else’s OS that does do that. I don’t want to have to buy third party programs to provide services that are logically operating system services. I’ve been there, done that, and all I have to show for it is the T-shirt :).

    M. Simon,

    I don’t care if Microsoft and Intel are “in cahoots” as you say, because I get a cheaper and more useful system from their being “in cahoots”.

    If you can make an equally useful system (defined by me, the customer) with fewer bytes, by all means put it on the market.

  12. Charles D. Quarles,

    There is a vast difference between an OS providing support for peripherals like the internet and in hardwiring that into the core OS itself.

    For example, MacOS X provides the Webkit framework which lets any developer include a fully functional web browser into their application with about 10 minutes of programing. I don’t have a problem with that. On windows however, Microsoft integrated IE into the file explorer itself creating an opportunity for web based malware to access the file system directly. There was no great clamoring from any customers to link the web browser to the file system. Microsoft did it for marketing reasons. They wanted to make their web browser the standard by so tightly integrating it into the system that not using it would be more trouble than doing so. (This all came out in the anti-trust trail). I think they were perfectly in there rights to do so but that doesn’t mean I have to approve of it from a design standpoint.

    Every time Microsoft folds more functionality into core OS itself it increases the complexity of the OS and that in turn increases the cost, development time, bloat and security problems.

    My point is that there is a real cost to Microsoft customers in all this interdependence. So far, the cost has not outweighed the benefits but if Microsoft cannot improve the situation it eventually will. The problems that Microsoft has had in shipping Vista (now 3 years overdue) is exhibit A. l The software tries to do so much and be so interdependent that it is choking on its own complexity. The industry press is full of stories of internal dissension and high level personnel changes all of which are indicative of a project in trouble.

    Microsoft’s market dominance gives it tremendous inertia but that inertia won’t protect them forever if they cannot manage the complexity of their own products.

Comments are closed.