A fascinating post by Wretchard on the dynamics of public events in the Internet age, and on the ways in which such events are now subject to quantitative analysis of the type that has previously been reserved for quantum systems and securities markets:
Internet storms are emergent events which are difficult to predict. They are like rogue waves on the ocean, arising from the complex interaction between many factors, none in themselves particularly threatening. Yet combined they can suddenly throw up a devastating phenomenon, able to sweep all before it. About all people can do to gain a semblance of influence over emergent events is to shorten their reaction times to events. In the jargon of the trade they must increase the speed of their feedback loops to have any hope of evading the avalanche or deflecting it decisively. Because there is no easy way to predict what direction emergent events will take, the prudent manager must do all he can to detect them while they are building up. A number of methodologies exist to do this. But perhaps the most simple consists of an analyst trained to look at prediction markets, aggregators and sentiment analysis software in ways designed to detect the edge of the storm.
How to integrate this new information in our understanding of public events, especially political campaigns?
I went over the Hillary/Internet storm problem earlier tonight with an analytics specialist and the discussion touched on some of the issues. Some of them were nearly philosophical. What should one look for? What, in fact was an “event”? But in general we concluded that simple analytical tools might make a difference in anticipating a sudden rising of the memes. Politicians might even be interested in using those tools one day. One day. But for Hillary, as for Dan Rather, it may already be too late.
Read the whole thing. There are also some insightful comments, including this one by geoffb:
My take is that there are often two opposing “opinions” that we have in us. The private one that we actually believe to be true and the public one that we think is what everyone else holds or is the PC one that is correct to express. The public one is molded by the media and those who hold power over our lives. It is what is captured by opinion polls.
The internet allows for the expression of the private opinion without the real world consequences. When enough people see that others believe as they privately do then they will lose the inhibition about expressing that private opinion and the private flips to be the public opinion.
See also this comment by Buckhead:
My take is these internet storms can only occur when there is a large difference between the cultivated image and the reality, analogous to a large potential difference in electricity. The situation is precarious, unstable and ripe for a spark or a leader (in lightning terminology) that will propagate a sudden discharge and restoration of equilibrium. In these internet storms that would be a closer relationship between image and reality.
Hillary’s invincibility and inevitability as a candidate has always been pure myth cultivated by the MSM and the Clinton machine. In truth, she has always been a deeply flawed candidate. She is an unappealing, obviously phony, deeply cynical shrewish harridan with the most grating voice in public life besides Gilbert Godfrey. The more she and Bill were in the public eye the more people were reminded why they didn’t like them.
In the case of Hillary, Image and reality have become more aligned over the course of a few days.
It doesn’t look like big-media and political professionals have even come close to appreciating what’s going on, or the full potential of the Internet and prediction markets. Many of the old-line experts still rely on traditional polling, and if they follow prediction markets they don’t admit it publicly. Yet while old experts are blindsided by events, market-savvy amateurs, using the Internet and thinking like traders rather than political wonks, seem to understand this stuff intuitively.
The Internet may have, in just a few days, destroyed the Clintons as a political force. It remains to be seen who among our political classes has learned the new rules. Maybe Obama has, maybe not. 2008 will be interesting.
UPDATE: Justin Wolfers writes about the prediction markets and the election. (via Chris Masse)
UPDATE2: Tom Maguire expands on his comments here.
My take is these internet storms can only occur when there is a large difference between the cultivated image and the reality, analogous to a large potential difference in electricity.
Geez, almost like someone calling out “The emperor has no clothes!” Maybe someone could turn it into a parable.
The Internet may have, in just a few days, destroyed the Clintons as a political force.
Sure, it may have. But neither you nor the other authors described any actual mechanism by which that happened, unlike in the Dan Rather saga they mention, where specific bloggers and freepers introduced new evidence and re-directed the debate.
In the Hillary case, you might just as well say that Iowa voters rejected Hillary and the MSM, which loves Obama and loves a David v. Goliath story, took up the cudgels to finish the job.
Suggested topic for evaluation – check out the Dean collapse in 2004. At one point he was at 90% in the prediction markets and the internet loved him – was he a victim of an Internet storm, or did reality bite?
Second topic – check the internet mentions of “Katrina” and “hurricanes” in late August/early September 2005. Now tells us whether an Internet storm swamped New Orleans, or simply described on an external reality.
The dog barks; the caravan passes by. I may post on this later, but honestly, this confusion of cause, effect, and commentary is embarrassing.
I don’t see what is embarrassing here. We are speculating about causes and effects. We don’t necessarily need to know the mechanism in each case, as long as we can find robust relationships between independent and dependent variables. Wretchard speculates that the amalgam of our political system and networked communications has emergent properties, which by definition means that the values of dependent variables will be difficult to predict by analyzing processes. Of course his hypothesis may be wrong, but the question is empirical. He suggests a way to test it, by observing prediction markets and sentiment indicators. If his hypothesis is wrong, his tests will not accurately predict political outcomes.
His testing method makes perfect sense to me, since it’s analogous to methods used to test quantitative trading systems in securities markets. You either make money or you don’t, and in this case you either predict political outcomes accurately or you don’t. There may be other ways of trading securities or making political predictions, that involve detailed understanding of processes, but if such ways exist their existence does not invalidate successful prediction systems that do not rely on understanding processes.
We are speculating about causes and effects. We don’t necessarily need to know the mechanism in each case, as long as we can find robust relationships between independent and dependent variables.
Well, I am certain that I could track the Tradesports futures market on the contract which pays off if the Tampa Bay Bucaneers win the Super Bowl, and run those odds against the internet chatter for the phrases “Eli Manning, poised quarterback”.
My as-yet-untested (but incredibly plausible) prediction – the odds on the Bucanneers collapsed just as the use of the phrase “Eli Manning, poised quarterback” soared.
Now, do you suppose that the internet chatter led to the defeat by the Giants (led by Manning) of the Bucanneers? Were the Bucanneers a victim of an internet storm?
Or would a search for causal mechanisms be useful? My guess is that such a search would strongly suggest that the Giants coaches and players defeated the Bucs *and* sparked the chatter.
If Wretchard can’t suggest any mechanism at all by which the internet chatter cratered the Hillary campaign, and can’t even establish cause-effect (for all we know, the big drops in the futures contracts *followed* the release of new polls supporting Obama), then all he has shown is that chatter will follow an event. That is an important insight?
And left out entirely – the big media (NY Times, for example) said the same thing that bloggers did, namely, Hillary looks a lot less inevitable after Iowa (hardly a radical or imaginative insight). Why couldn’t her collapse have been entirely a MSM phenomenon, with no role played at all by the blogs? That at least would be an interesting speculation, and might be true.
So, Wretchard is missing any cause and effect, and is missing any attempt to display that the internet message differed from the MS message. Yet he concludes that “the destruction [of her campaign] was accomplished through the agency of an Internet storm.”
Unimpressive.
And I have no idea why this has irked me so – normally I find Wretchard to be fine, and have always admired the Chiccago Boyz.
You may be correct about Hillary and Wretchard may be incorrect — maybe this was no Internet storm. However, I think that you are confusing the specific question of Hillary with general questions about how information influences events.
The problem in your football hypothetical is the same as the problem with the hemline theory of stock market performance, and with other bogus theories based on spurious correlations. They don’t make sense, which was your point, and you (generic you, not TM) can easily fool yourself if you are not careful in analyzing the data. But that doesn’t mean that all such correlations are spurious, and if you can find a way to test them it may turn out that some of them have predictive value.
People test ideas in this way all the time in financial markets. Why should politics, another area of life that runs on crowd psychology, not be a fertile area for similar research? Wretchard sees this possibility and so do I. It may be that research into such issues will ultimately prove fruitless, but it’s an empirical question, so why not look into it?
I think to grasp what Wretchard is talking about, one has to compare the current mode of mass debate with that of the pre-internet era.
Look back to circa 1970. Back then, probably fewer than a hundred people controlled what issues reached a mass audience. Three major news networks and a handful of major papers set the terms of the debate in stone. The editors of those organs dictated what did and did not get discussed. Those gatekeepers created a interrupt in the feedback loop. A small idea or event could propagate and spread only if gatekeepers wished it. Basically, if something didn’t seem significant to a few dozen left of center white guys in New York, then no one in the wider world ever heard of it.
Mathematically, the properties of this system resemble a beaker of water at room temperature. The actions or opinions of one water-molecule/citizen could not perturb the total system because changes in the state of one molecule/citizen did not propagate across the entire system. Fluctuation in temperature/opinion dampen out toward the norm. Simple statistical tools allowed one measure and predict the future behavior of the system.
The internet changed this dynamic. Without the gatekeepers to dampen out the fluctuations, the system begins to mathematically resemble a beaker of water a critical freezing or boiling temperature. Small fluctuations get amplified and spread across the entire system. A minor cluster of low temperature water molecules caused by simple chance triggers a beaker of critical water to freeze solid by creating a self propagating feedback look of freezing water molecules. A single poster on Free Republic triggers a feedback loop as individuals spread the word that something in a major news story in suspicious.
The killer here is that if we are entering a period of critical/emergent mass debate, then the course of events governed by such debates might be fundamentally unpredictable because they hinge on the amplification of fluctuations so small as to be practically unmeasurable.
(Tom Maguire’s post on this topic is here.)