Policy, Anecdotes and The Problem of The Black Swan. Why Events Like Comcast/Netflix and Fire Island Matter.

Often in policy debates I find myself facing a broad general statement, such as “Wireless is just as good for everyone as wireline, just look at how the market has adopted it.” Or “ISPs would never block or degrade service because they would lose customers.” Point to a counter example, e.g., “Verizon’s effort to replace wireline with Voicelink on Fire Island was a total flop” or “But Comcast, AT&T, Verizon and other ISPs have deliberately allowed Netflix quality to degrade as a negotiating strategy” and the response is invariably “Oh, that’s just an anecdote and you can’t base rules on anecdotal evidence.”

 

Oddly, this throws most people into a tizzy of confusion because (a) they vaguely remember learning something about anecdotes not being proof or something; (b) everyone always says anecdotes aren’t proof; but (c) the general statement is clearly false based on real world experience. People know that “it’s only an anecdote, therefore it doesn’t count” is a bull$#@! answer, but they can’t explain why. Hence confusion and much bull$#@! going unchallenged in policy.

 

In logic, we refer to this as “The Problem of the Black Swan.” No, this has nothing to do with the somewhat racy but very artsy so that makes it OK movie starring Natalie Portman. And, while it is the inspiration for the book by Nassim Nicholas Taleb, it actually means something different. “The Problem of the Black Swan” is a demonstration of the problem of reasoning by induction and falsifiabilty. You cannot prove all swans are white just by finding a white swan, but you can disprove all swans are white by finding a single black swan.

 

While I don’t normally use this blog to teach Logic 101 type stuff, application (and misapplication) of the “Problem of the Black Swan” comes up so often that I will delve into this below. By the time we’re done, you will be able to explain to people who pull that “oh, an anecdote isn’t evidence” crap exactly why they are wrong. You’ll also be able to apply the “anecdote rule” properly so that you don’t get caught in any embarrassing errors.

Elucidation below . . .

 

What Is ‘The Problem Of The Black Swan?’

 

Here’s the problem. I want to make a very broad general statement about the universe such as “all swans are white.” But the only evidence I have for this rule is observation, i.e., the fact that every single swan I see is white. In Europe, this was true for thousands of years. No one had ever seen a swan that wasn’t white. Reasoning by induction (i.e., generalizing a rule from all observed cases), I conclude that “all swans are white.”

 

Along comes a fellow holding a black swan. This single black swan shows that the general rule “all swans are white” is false. I don’t get to say “that black swan is just an anecdote so I don’t care” and pretend it doesn’t exist. The black swan is real and sitting in front of me. Even if I previously observed a million white swans, and only white swans, in all your recorded history, the single black swan disproves the general statement “all swans are white.” Because a general statement like “all swans are white” is only true if every single swan is white.

 

The Problem of the Black Swan shows the limits of inductive reasoning. I can never definitively prove something via inductive reasoning. As Europeans discovered when they reached Australia and discovered, after several thousands of years of assuming all swans were white, that there were indeed black swans.

 

How This Works With Anecdotes and Policy.

 

Applying the Problem of the Black Swan to policy arguments, we can now explain how to call out bull$#@! as it usually arises. Let us return to the following example I run across regularly.

 

General Statement. Millions of Americans have totally dropped their landline for wireless. In fact between 40-45% of the market has dropped landline service entirely. This proves we can eliminate landline service without worrying about bad consequences.

 

Black Swan. Superstorm Sandy destroyed landline service on Fire Island. In 2012, Verizon attempted to replace landline service with a wireless alternative called VoiceLink. Residents of Fire Island hated it and complained of numerous problem with the service. Verizon ultimately agreed to install FIOS as an alternative to VoiceLink.

 

Application of the Black Swan To Policy. Fire Island falsifies the statement “we can eliminate landlines with no problems.” The fact that it is a single case does not change the fact that it falsifies the general statement. This is an application of the black swan problem. You can’t waive it away by claiming it is “anecdotal” because it friggin’ happened and therefore it could happen again.

 

The Limits of Anecdotes: A Single Anecdote Does Not Prove A Generalization.

 

It is important to recognize the limitation of the black swan. It is equally false to say “Fire Island shows you can never eliminate landlines and replace them with wireless.” Here, the general statement from induction is “any effort to eliminate landlines and replace them with wireless will look like Fire Island.”

 

The general statement “wireless cannot replace wireline” would be falsified by a successful total replacement of wireline by wireless. This is why the AT&T proposed wireline to wireless trial is important. If AT&T demonstrates it can successfully replace a wireline network with an all wireless network without degrading service to subscribers, it will provide a black swan that falsifies the general statement “all efforts to replace wireline with wireless are total disasters like Fire Island.”

 
A Single Case Falsifies A General Proposition, But Does Not Go To the Merits Of The Argument.

 

Another point of confusion is what the black swan disproves. The black swan does not tell us how to resolve the merits of an argument. It simply tells us that a general proposition is false. For example, consider the following from Comcast’s application to acquire Time Warner Cable.

 

General Statement. Comcast argues that ISPs would never willingly degrade applications, particularly popular applications like Youtube and Netflix, to extort payments or to disadvantage a competitor for video programming. If an ISP tried such a tactic, it would lose customers. Therefore, Comcast argues, we don’t need to worry about it buying TWC because no matter how big Comcast gets, or how much online video threatens its traditional video market, Comcast cannot degrade rival video traffic.

 

Black Swan: As part of a negotiation over rates, Comcast refused to upgrade its peering links with Netflix’s middle mile providers. This had the effect of degrading Netflix’s service to Comcast’s customers. Comcast, and other major ISPs such as Verizon and AT&T, employed this tactic for months until Netflix agreed to pay for direct interconnection. Once Netflix and Comcast reached an agreement, Netflix performance on Comcast’s system improved dramatically.

 

Nor was this secret. Netflix published regular reports on how service continued to degrade on specific ISPs. Despite this, customers did not switch to a different ISP (nor is it clear it would have made a difference, as AT&T and Verizon, the primary competitors of Comcast, were engaged in a similar negotiating tactic and suffering similar degradation in Netflix performance as a consequence). Instead, according to Netflix, Netflix began to suffer customer loss and ultimately felt compelled to enter an agreement with Comcast.

 

Application of the Black Swan To Policy. The debate here generally devolves into something utterly irrelevant about the black swan and something the black swan cannot tell us – the merits of the policy argument. But the black swan doesn’t tell us anything about merits. Just as the appearance of a black swan does not tell us if white swans are somehow ‘better’ than black swans, the Comcast/Netflix fight does not necessarily prove anything about whether ISPs are right to demand payment or if Netflix is right that this is an exercise of market power by the largest ISPs. To answer that question, we need our more conventional tools of policy analysis.

 

What the black swan does is falsify a general statement. Here, the general statement is: “ISPs can never degrade service to an application, particularly a popular application such as Netflix, without losing customers.” Whether or not Comcast and other ISPs are right in the merits of their argument is irrelevant to the fact that the general statement is now proven false. ISPs can allow service to a popular application to degrade, publicly, for an extended length of time, without suffering customer loss.

 

Accordingly, a policy based on the proposition that ISPs can “never” engage in such behavior and therefore we don’t even need to worry about whether ISPs have market power – either through size or through use of their “termination monopoly” – is based on a demonstrably false premise. The fact that this has only occurred once (twice if one counts Comcast/BitTorrent) is irrelevant. The occurrence of the black swan (Comcast/Netflix) falsifies the general statement that all swans are white (that ISPs can never degrade service of popular applications without losing customers, rendering rules unnecessary).

 

In other words, you don’t get to make the black swan disappear by simply declaring it an “anecdote” or “one time event.” While the occurrence of the event doesn’t tell me the answer, it does tell me I need to actually address the issue rather than conclude it could never happen in the first place. Put another way, you cannot claim an event can never happen when it has actually happened.

 

So Where Did This Whole ‘Anecdotes Aren’t Data’ and ‘We Don’t Make Policy Based on Anecdotes’ Nonsense Come From?

 

When we talk about the difference between broader quantitative studies and “anecdotes,” we generally mean a bunch of different things. Usually it has to do with trying to establish a relationship between two things when I observe something that suggests a relationship (or have a theory that suggests a relationship) between things – particularly in a complex environment.

 

For example, the fact that one person who smokes gets lung cancer does not prove with any degree of certainty that smoking causes cancer. A lot of things might have caused the lung cancer. Similarly, the fact that one person who smokes lives to be a hundred and dies of a heart attack during sex does not disprove a relationship between smoking and cancer. This person might be unusually resistant to lung cancer.

 

But if I keep track of hundreds of thousands of smokers and non-smokers, and control for exposure to other potential cancer causing elements, I can determine whether there seems to be a relationship between smoking and lung cancer. I can also get some idea as to how much of a relationship there is between smoking and lung cancer. For any single individual, there may be multiple possible causes of lung cancer. But if I aggregate enough individual data and analyze it properly, I can start making tentative (and, over time, increasingly less tentative) conclusions about the relationship.

 

The same thing is true for complex phenomena like global climate change. Here, I start with a theory – the increase in concentration in the atmosphere of certain kinds of gases (“greenhouse gases” such as carbon dioxide) traps heat, altering Earth’s climate over time. OK, that’s a nice theory. Does it bear out in practice? Here again, a single weather event doesn’t tell me much. Why? Because I cannot prove or falsify the theory by a single weather event. The theory is broad enough to encompass both an exceptionally hot summer or an exceptionally cold winter because weather false within a range and we can have extreme events (“outliers”) that don’t fit the pattern. In cases like this, I need to demonstrate the validity of the theory by making testable predictions and by gathering sufficient data to show the correlation between the build up of the greenhouse gases and the predicted changes in global climate.

 

The “we don’t make policy based on anecdotes” idea also comes up in cost/benefit analysis. Suppose the odds of something happening seem very small. Not that it never happens, but that it seems extremely unlikely to happen or only happens very rarely. Do we still need a rule or procedure to address the rare case? That depends on a number of factors, including the possible severity of the consequences. If my wireless call drops a bunch of times, I generally accept that as reasonable provided I can still use the phone. But when 9-1-1 geolocation fails, or if wireless becomes my only possible means of communication because the phone company has eliminated the only existing wireline network, I get a lot more concerned about the “anecdotal” cases.

 

Conclusion

 

When we debate policy, we get a lot of misdirection and slight of hand. When a skilled illusionist like David Copperfield uses misdirection and slight of hand to make something disappear, it’s an amusing trick. When lobbyists use misdirection to make an inconvenient truth disappear, it stops being amusing.

 

When something happens in reality that disproves a general statement, you don’t get to wish it away by declaring it an “anecdote.” The next time someone pulls that tactic on you, remember the Problem of the Black Swan. And while we normally consider it rude to explain how a magician does his tricks, feel free to call bull$#@! on lobbyists trying to make reality disappear.

 

Stay tuned . . .