Institutions, Instruments, and the Innovator’s Dilemma

I have written several posts that use Carroll Quigley’s “institutional imperative” as a lens for understanding contemporary events. [1] Mr. Quigley suggests that all human organizations fit into one of two types: instruments and institutions. Instruments are those organizations whose role is limited to the function they were designed to perform. (Think NASA in the 1960s, defined by its mission to put a man on the moon, or the NAACP during the same timeframe, instrumental to the civil rights movement.) Institutions, in contrast, are organizations that exist for their own state; their prime function is their own survival.

Most institutions start out as instruments, but as with NASA after the end of the Cold War or the NAACP after the victories of the civil rights movement, their instrumental uses are eventually eclipsed. They are then left adrift, in search of a mission that will give new direction to their efforts, or as happens more often, these organizations begin to shift their purpose away from what they do and towards what they are. Organizations often betray their nature when called to defend themselves from outside scrutiny: ‘instruments’ tend to emphasize what their employees or volunteers aim to accomplish; ‘institutions’ tend to emphasize the importance of the heritage they embody or even the number of employees they have.

Mr. Quigley’s institutional imperative has profound implications for any democratic society – especially a society host to so many publicly funded organizations as ours. Jonathan Rauch’s essay, “Demosclerosis” is the best introduction to the unsettling consequences that come when public organizations transform from instruments into institutions. [2] While Mr. Rauch does not use the terminology of the Institutional Imperative, his conclusions mesh neatly with it. Describing the history and growth of America’s bureaucratic class, Mr. Rauch suggests its greatest failing: a bureaucracy, once created, is hard to get rid of. To accomplish whatever mission it was originally tasked with a bureaucracy must hire people. It must have friends in high places. The number of people who have a professional or economic stake in the organization’s survival grows. No matter what else it may do, it inevitably becomes a publicly sponsored interest group. Any attempt to reduce its influence, power, or budget will be fought against with ferocity by the multitude of interests who now depend on it. Even when it becomes clear that this institution is no longer an instrument, the political capital needed to dismantle it is just too high to make the attempt worth a politician’s time or effort. So the size and scope of bureaucracies grow, encumbering the country with an increasing number of regulations it cannot change, employees it does not need, and organizations that it cannot get rid of.

I used to think that the naked self-interest described by Mr. Rauch was the driving force behind the Institutional Imperative. It undoubtedly plays a large role (particularly when public funds are involved), but there are other factors at play. One of the most important of these is what business strategists call Marginal Thinking.

Clayton Christenson, professor of Harvard Business School, explains the problems of marginal thinking in his book, How Will You Measure Your Life? He introduces the topic by relating the demise of Blockbuster, king of the traditional video rental stores, at the hands of Netflix, then an unknown upstart.

“A little upstart called Netflix emerged in the late 1990s with a novel idea: rather than make people go to the video store, why don’t we mail DVDs to them? Netflix’s business model made profit in just the opposite way to Blockbuster’s. Netflix customers paid a monthly fee—and the company made money when the customers didn’t watch the DVDs they had ordered. As long as the DVDs sat unwatched at customer’s homes, Netflix did not have to pay return postage—or send out the next batch of movies that the customer had already paid the monthly fee to get.

It was a bold move: Netflix was the quintessential David going up against the Goliath of the movie rental industry. Blockbuster had billions of dollars in assets, tens of thousands of employees, and 100% brand recognition. If Blockbuster decided it wanted to go after this nascent market, it would have the resources to make life very difficult for the little start up.

But it didn’t.

…When it compared Netflix’s numbers to its own, Blockbuster’s management concluded, “Why would we bother?” The market Netflix was pursuing was smaller; it might get bigger, but it was unclear how big it had the potential to be. More troubling for Blockbuster’s management, though, was that Netflix’s profit margins were substantially smaller than what Blockbuster was used to. If Blockbuster did decide to attack Netflix, and if it was successful, those efforts would most likely cannibalize sales from Blockbuster’s very profitable stores….

Netflix, on the other hand, thought this market was fantastic. It didn’t need to compare it to an existing and profitable business: its baseline was no profit and no business at all. Compared to that, Netflix was very happy with their relatively low margins and their “niche market.” [3]

Americans know what happened next: the niche market became the main market and Blockbuster lost all of its business. It declared bankruptcy in 2010. Mr. Christensen comments on the flawed principles behind Blockbuster’s disastrous strategy:

“Blockbuster followed a principle that is taught in every fundamental course in finance and economics: that in evaluating alternative investments… base decisions on the marginal costs and marginal revenues (the new costs and revenues) that each alternative entails.

It’s a dangerous way of thinking. Almost always, such analysis shows that the marginal costs are lower, and the marginal profits higher, than the full cost. This doctrine biases companies to leverage what they have put in place to succeed in the past, instead of guiding them to create the capabilities they will need in the future. If we knew the future would be exactly the same as the past, that approach would be fine. But if the future is different—and it almost always is—then it is the wrong thing to do.

Blockbuster and Netflix revenue, 2004-2010
Image source

Blockbuster looked at the DVD postal business using a marginal lens: it could only see it from the vantage point of its own existing business. When viewed like this, the market Netflix was going after did not look at all attractive. Worse, if Blockbuster did go after Netflix successfully, this new business was likely to kill Blockbuster’s existing business. No CEO wants to tell shareholders that he wants to invest to create a new business that is going to be responsible for creating existing businesses, especially if it is more profitable.

[Thus] Blockbuster believed that the alternative to not pursuing the postal DVD market was to happily continue doing what it was before, at a 66% profit margin. But the real alternative to not going after Netflix was, in fact, bankruptcy. The right way to look at this market was not to think, “How can we protect our existing business?” instead, Blockbuster should have been thinking: “If we did not have an existing business, how could we best build a new one? What would be the best way for us to serve our customers?” Blockbuster couldn’t bring itself to do it, so Netflix did instead. And when Blockbuster declared bankruptcy in 2010, the existing business that it had been so eager to preserve by using marginal strategy was lost anyway.” [4]

Blockbuster fell victim to the institutional imperative. Its management forgot that a good business aims to accomplish “a job [that needs] to get done,” not milk existing assets. [5] Blockbuster focused on preserving its institution at the cost of becoming the instrument consumers wanted. Consumers chose the instrument over the institution. We can glean a few insights from this account of marginal thinking and disruptive innovation that can be applied outside the business world.

In environments of real competition, institutions lose out to instruments. If we want to free our society from the burden of rent-seeking institutions and fill it with instruments useful to humanity, then we must do everything that we can to ensure that the environments in which organizations do their work are competitive. As the story of Blockbuster and Netflix suggests, the marketplace is an ideal environment for the contest of instruments and institutions; the pressures of the free market are hostile to the existence of firms who have lost their instrumental value. This is the genius of the market system: it forces decrepit institutions to face “the innovators dilemma” or fall to the wayside.  It is also why state intervention in the system can be so destructive. Economic policies or government action that reduces competition – either by subsidizing its winners or by bailing out its losers – will inevitably lead to the creeping “institutionalization” of free enterprise. [6]

Applying these principles to government bureaucracies, regulatory agencies, or other organs of the state not subject to market pressures is more difficult.[7] In an age of fast passed disruptive innovation – particularly on the part of terrorists and other organizations that pose a national security concern – this is a serious matter. Those willing to consider the question face a two-pronged problem:

  1. How do we discourage governmental organizations from adopting “marginal strategies” when their instrumental usefulness begins to fade?
  2. .

  3. How do we create the type of competitive environment for publically funded organizations that will be just as hostile to “institutions” as the market is? Is creative destruction possible in the public sphere?

This essay was originally published on The Scholar’s Stage on 1 March 2013.


[1]  T. Greer. “Dreaming Grand Strategy” The Scholar’s Stage. 12 May 2010.
And T. Greer. “The Death of a Nation.” The Scholar’s Stage. 30 January 2010.

[2] Jonathan Rauch. “Demosclerosis” National Journal. 5 September 1992.

[3] Clayton Christensen, James Allworth and Karen Dillon. How Will You Measure Your Life? (New York: Harper Collins Publishers). 2012. pp. 179-180

[4] Ibid. pp. 181-183

[5] This is another idea Clayton Christensen likes to stress. See Carmen Noble, “Clay Christensen’s Milkshake Marketing.” Harvard Business School: Working Knowledge. 14 February 2011.

[6] See Ashwin Parameswaran, “Cause and Impact of Crony Capitalism: The Great Stagnation and the Great Recession“, MacroEconomic Resilience, (24 November 2010) for a longer (and denser!) explanation of this problem in its current context. It is a regular theme for his work; check the tag ‘cronyism’ for his more recent posts on the subject.

[7] I realize that the simplest solution to this problem is to not create organizations dependent on public funds, especially if they compete directly with private organizations. I will leave the discussion of which organizations rightly belong in the government and which ones do not for a different day. Whatever your opinion on the matter (save perhaps if you are the rare anarchist) we can agree that it is in our best interests to ensure that any organization the government creates (be it a research department, regulatory agency, foreign office, or military command) is an instrument, not an institution.

16 thoughts on “Institutions, Instruments, and the Innovator’s Dilemma”

  1. Quick note on this essay: I have thought much about this question after I wrote this essay several months ago. I have come to the conclusion that decentralization is one of the best ways to remedy this problem, and briefly explained these thoughts at the end of this post. Since then I have become aware of similar proposals by Arnold Kling and of course the excellent material on decentralization found in America 3.0. Last month the Stanford Social Innovation Review published an essay by Clayton Christenson and a few others presenting the research they have conducted on this topic over the last year. (Their focus is not on decentralization and creative destruction, but incentive structures).

    All of these materials may help us come closer to an answer to the two questions posed at the end of this post. Suggestions by the readership are welcome.

  2. Another example is the closing of the Sears catalog store just as Amazon was ramping up their business. Sears began as a catalog business and moved into retail stores later. In the 1890s you could buy a house from Sears and have it shipped to your location. By 2000, they had no idea where to go.

    Eastman Kodak had a worse problem because digital photography had no alternative for profits by Kodak. They would have had to start an entirely new business.

    Nice post. I’m going to read the linked articles.

  3. Incidentally, I heard a rumor yesterday from someone who should know, that a Memorandum of Understanding has been submitted to the Canadian equivalent of the FDIC about a very large bank that may be going under. It could be CITIbank according to my source. Crony capitalism doesn’t always work out.

  4. Hmm. Hearing that, I’m very glad that my mortgage is with another national bank — and that my daughter was able to convince me to switch over from B of A to a Texas bank which has been very circumspect in conducting itself over the last couple of decades.
    Very nice, customer-service oriented bank, by the way. I can’t say enough good about them.

    Sears so seriously missed the boat, in not adapting in time to go toe to toe with Amazon. They were truly the king of catalog sales in the day; some of the nicest old houses that I know of in various small towns in Texas were Sears catalog homes. Amazon could seriously use competition, just to keep them on their edge.

  5. I worked for Sears for a while in the 1950s and they were abysmal in data handling. They could not have figured out how to use the internet. Too bad but they became an institution. So many employees had invested in “profit sharing” and had large accounts that some of them could not be fired. They were big stock holders. I wonder what happened to them. I have a cousin who is 94 and she has been living on her Sears shares since her husband died 40 years ago.

  6. Sgt Mom – “Craftsmen Homes” – kit homes in the Bay Area – are going for over $1 million these days – Sears missteps – reminds me of the classic issue of the railroads “missing the boat” in the early 1900s by thinking of themselves as “railroad” companies and not “transportation companies:

  7. Bill, if only Sears had hired Peter Drucker for a week. I tried to get him for a dap at the medical association when I was president of it. That was 1986-87. His fee was $10,000 for an hour or a day. We couldn’t afford him. Too bad. We were trying to figure out how to reform medicine at the time.

    The medical association in Phoenix, Maricopa County, tried to do a program for the poor and lower middle class by getting docs to accept lower payment. The idea was to establish a panel who agreed to lower prices. The Federal Trade Commission came down hard on them and they were almost bankrupted by it. No association could fix fees even if it was to keep them lower ! The dissent invoked Broadcast Music, another Supreme Court case but it was a minority opinion. That ended doctor driven health reform. The left, of course, won’t admit that.

    The case is FTC vs Maricopa County Medical Association.

  8. Mike K, your experience confirms one of Christenson’s main points – not only is an entrenched incumbent unwilling to change, but they are unable. Even when they fully understand the disruption and its implications, they can’t do anything about it.

    I came across this list recently

    Companies that survive are small and are usually service oriented

  9. }}} I have come to the conclusion that decentralization is one of the best ways to remedy this problem

    So, like, keeping the big overarching government SMALL and breaking the government up/down into, oh… let’s call them states, so that each one is a laboratory for ideas — and only after it’s proven successful does any idea spread around widely…?

    Hey. What a concept! We should have done that centuries ago!!



  10. }}} Very nice, customer-service oriented bank, by the way. I can’t say enough good about them.

    Well, the name of them would be a real good start :-D

    }}} Companies that survive are small and are usually service oriented

    Considering that it has become increasingly obvious since the 1960s that the real term for a “post-industrial economy” is an “IP & Services Economy”, this seems… obvious.

    Offhand, the largest service orient business that does spectacularly well is the Florida-based Publix supermarkets. The company seems to do an amazing job with keeping the Right Attitude regarding their employees and policies, even though they’re, in many cases, gaining monopoly privelege within the local supermarket community. In Florida, they now sometimes outnumber ALL local competitive stores together something like 3 and 4 to 1.

    It’s possible for a large company to thrive and prosper under an IP&SE, but it’s difficult, since large companies attempt, usually, to go hierarchical, and that’s exactly the wrong model for an IP&SE organization.

    The best model for them (I argue) is a loose network of working groups, with some specialization but not a tremendous focus on it — if a working group needs a skill it should have a company pool to draw from, and make up its groups ad hoc as much as possible. This helps keep the people loose and focused on the problems and issues at hand rather than on a specific function.

    The problem with a hierarchical model is that info has to flow up, then down, to get from point a to b in the inverted tree, which means that any number of individuals can become a bottleneck — either deliberately (as shown in many Dilbert cartoons, with someone seeing their power coming from controlling information others need), or inadvertently (when someone doesn’t realize the importance of their knowledge and/or are just flat out clueless or incompetent in their job). In an IP&SE this blockage can be downright deadly.

    I’d bet more than one lower level exec at Blockbuster could see the writing on the wall long before Blockbuster started to fail… but they had no authority and couldn’t even TRY to do any such sort of service.

    A network model is, as suggested, decentralized, with, ultimately, groups that find/pick their own issues to deal with and let others know they’re on them. About the only centralized element is keeping such groups focused on what the company perceives as its business. It is flexible and versatile.

  11. “this seems… obvious”

    Service oriented
    and also consumer oriented in markets with a lots of competition.
    This allows for constant, immediate feedback from customers/patrons/tenants resulting in rapid evolution or destruction.
    Network infrastructure of competitors & consumers creates positive feedback loops with lower transaction costs for labor and innovation and information flows.

    Obvious to observe maybe, but for various reasons not easily realized.

  12. I don’t think we can say that Blockbuster made the wrong decision.

    The course they followed, conscious or not, was to embrace the creative destruction of capitalism. With the advent of NetFlix, a rational decision could have been to milk their changing market, distribute the profits, then liquidate through bankruptcy. Blockbuster didn’t have a huge physical plant that couldn’t be closed. Store leases can be gotten out of, and inventory turns over rapidly anyway. The bulk of employees were minimum wage people anyway with no special skills.

    Investors could have taken their Blockbuster dividends and re-invested them where they may, including in NetFlix.

    Governments are, of course, another problem. However, small units, like cities, do vanish at times.

    Too bad we don’t a bankruptcy court for the IRS!

  13. Individuals face these dilemmas too. A typical example: Do you leave the city where you’ve built a career for a promising opportunity in another city? Most people will not do it unless things are going poorly for them in their current careers and they don’t see better prospects locally. IOW they behave as Blockbuster did.

    There is an element of selective hindsight in the Blockbuster parable. We see Netflix as a threat to Blockbuster because Netflix became a big success. We don’t see the upstart competitors to Blockbuster that failed.

  14. @Jonathan –

    The “selective hindsight” problem is a particularly noxious pox afflicting the business book industry, especially those aimed at a popular audience. Clayton Christenson sets himself apart from this crowd through his dedication to theory. Christenson does not seek to find ‘just so stories’ from which to draw basic principles from, as most business writers do. His is a search for a theory – a theory of business innovation. He looks for a theory that can describe why small businesses overwhelm and out perform big – but smart – businesses. His theory applies to ‘disruptive innovation’ in the days of the big Oil trusts to the rise of Silicon Valley over the last few decades. Much better on than picking a few of the recent losers/victors and creating a ‘just-so-story’ for them.

Comments are closed.