Look! My Solution Found A Problem! Comcast Degrades BitTorrent Traffic Without Telling Users.

O.K., free speech issues are always sexier. Nothing gets the public (or me) wound up like blocking NARAL or censoring Pearl Jam. But, as Ecclesiastes tells us: “Money answers all.” (10:19) At the very least, it tends to rivet people’s attention without the distraction of whether or not you like the speaker or the message.

So I was quite pleased to see the Associated Press run this story on how Comcast degrades BitTorrent traffic in the name of quality of service (QoS), especially after Comcast had denied such rumors as vicious lies last August. (Where is Mona “the Hammer” Shaw when we need her?) While my friend Greg Rose on Econoclastic gives his (to my mind quite plausible) theory as to why Comcast would engage in such blocking on a large enough scale to be worth getting caught, I would like to play out the public policy implications of Comcast’s actions.

As I discuss below, this recent episode underscores several of the critical points I have made in the past about the economics of access, but without all the sexy free speech stuff clouding things up. In particular, I hope all those idjit content producers like Viacom that oppose Net Neutrality they think it will help police content for infringement and give them an advantage over rivals who can’t afford to pay the “fast lane fees.” Because, as Comcast’s little tepid step toward “How to Monetize Monopsony Power and Make the World Your Bee-Yatch” shows, making a deal with the broadband access devil to police your content guarantees that broadband access providers will end up owning you the way Microsoft ended up owning IBM and everyone else who thought that they could leverage another parties control of a bottleneck facility to its own advantage.

Given the amazing track record the IP mafia has for making bad decision in this regard, I’m not exactly holding my breath they will see reason. But I can at least secure myself the bitter pleasure of saying “toldja so” after it’s too late.

More below….

OK, just to make sure we understand what’s going on. The AP conducted some experiments that showed that Comcast interfered with subscribers trying to upload files via an application called BitTorrent. As described in the article, the AP attempted to upload a file of the King James Bible to Bittorrent servers from Philadelphia and San Francisco (attempts to replicate in Boston failed for unrelated reasons). The efforts to use Bittorrent were apparently blocked or degraded by Comcast inserting “reset packets” into the packet stream sent and received by the AP’s test computer using Comcast as its broadband access ISP. An analysis of traffic to a Time Warner server trying to download content from a Bittorrent “swarm” showed that half the reset packets came from computers connecting to the Internet via Comcast, and that only Comcast users consistently generated reset packets.

From this, and from control tests conducted using computers connecting through other networks, the AP concluded that Comcast was deliberately inserting reset packets into the bit stream sent from Comcast subscriber computers. In other words, the tests demonstrate that Comcast was deliberately, and without informing its users, degrading the ability of users to use Bittorrent. In response, a Comcast spokesman replied:

“Comcast does not block access to any applications, including BitTorrent. We have a responsibility to provide all of our customers with a good experience online and we use the latest technologies to manage our network.” Douglas also said that using various techniques to manage bandwidth is “standard practice for [Internet service providers] and network operators all over the world. We rarely disclose our vendors or our processes for operating our network for competitive reasons and to protect against network abuse.”

For purposes of this discussion, let us assume everyone here tells the literal truth. First, “Comcast does not block access to any applications, including BitTorrent.” Very true. What Comcast is doing here is far more subtle. It is discouraging use of BitTorrent by degrading the reliability of the application, not by out and out blocking. This certainly will impact the load of traffic on Comcast’s systems although, as we shall see, the impact is not limited to Comcast’s systems or Comcast’s customers. At the same time, we will also note that the AP used the King James version of the Bible. This file is (a) relatively small as things go (only about 4 MB) and, (b) clearly in the public domain. So this is clearly an effort to degrade BitTorrent use to reduce overall BitTorrent traffic, regardless of the size of the file or the content.

Now let’s play all this out.

Implications for Quality of Service. The most often used argument by opponents of network neutrality, and the usual excuse provided by Comcast and others caught red-handed doing this sort of thing, is that they need to do this to maintain a suitable level of quality of service. But as the Comcast conduct here indicates, there are many problems with this argument.

First, Comcast is not guaranteeing quality of service. Just the opposite. It is actively degrading service for subscribers, many of whom subscribe to high-bandwidth services precisely for the purpose of getting access to muckin’ big files (or, frankly, they’d stick to much cheaper dial up). So when Comcast here speaks of “a responsibility to provide all of our customers with a good experience online,” it really means “We are trying to screw up traffic of high-bandwidth users in a non-obvious way so we don’t have to make expensive upgrades or engage in in obvious metered pricing.” As I have written before, cable operators in particular face network capacity constraints because of the way they constructed their systems. So they are allowed to advertise their “always on, all you can eat” speed based on certain theoretical assumptions about network usage that are increasingly unrealistic.

Comcast’s basic problem here is it wants it both ways. It wants to advertise all you can eat connections of the highest speed — because that sells so much better than alternatives like metered pricing or explicit bandwidth caps. But it doesn’t want to deal with the consequences of the user behavior this sort of advertising generates. i.e., people using their Comcast connection all the time and expecting the advertised speed. Nor does Comcast want to deal with this the traditional way, by spending the money to build more capacity, then charging a higher price for the new “top speed.” Comcast, like any other profit-maximizing firm, would prefer to avoid expenditures and, if it must spend money to gain revenue, would prefer to minimize expenses and maximize revenue. Inserting reset packets to degrade the reliability of BitTorrent, and therefore discourage its use overall, is much cheaper than upgrading from existing hybrid-fiber-coax to fiber. So, as I predicted over a year and a half ago Comcast makes the logical choice and degrades traffic that eats bandwidth rather than pay to upgrade.

So calling this a means of maintaining QoS is a tad Orwellian, although utterly rational and consistent. Usually when people argue about maintaining QoS, they mean actually guaranteeing a high level of service for people who want it. Here, Comcast actually means degrading service for particular applications of its own choosing (without telling users) so it can maintain an acceptable best efforts level of service for everyone else without expensive investment.

From a public policy perspective, I would argue that this sort of “Quality of Service” is not what we wish to encourage because it is a “deceptive trade practice” and, more fundamentally, it “sucks rocks.” Indeed, even folks who oppose full on net neutrality, like Robert Atkins and Phil Weiser, think we should address this by requiring complete disclosure of speed limits, disfavored applications, or other such practices generally used to throttle traffic. That is to say, let Comcast or other ISPs degrade BitTorrent so long as they tell users they are doing so. Then users who care either won’t subscribe or will switch providers.

To keep the ball rolling here, I will ignore the practical problems with this assumption, such as why I think throwing even more roadblocks for broadband uptake into the systems creates real problems from a public policy perspective and whether people can really switch providers easily in response. Rather, let me move on to the next reason why allowing Comcast to engage in the kind of conduct demonstrated by the AP experiments is a a Bad Idea from a public policy perspective.

BitTorrent Is An Edge Based QoS Application; Killing It Creates More Problems. Saying that BitTorrent is bad because it accounts for 90% or so of Internet traffic (assuming that figure is accurate) is a lot like saying TCP/IP is bad because it enables 100% of internet traffic. Few people realize these days that BitTorrent is not some provider like Napster out there to help people swap copyrighted material. Rather, BitTorrent is an open source, edge-based application that helps people move large files around by breaking them into small files. In plainer English, people came up with BitTorrent because they found the existing last mile (built by the likes of Comcast) way too slow and congested for the stuff they wanted to do. So they wrote an application that made the system work better and more efficiently.

Because BitTorrent works so well and efficiently, it became a favorite of people trying to move large files. This includes, no surprise, trying to move around video and audio files — especially video files like movies. And, in no small part because the content companies have done their best to keep people from easily buying and downloading their content legally for prices and on terms users find attractive, BitTorrent quickly became a favorite way to move pirated movies and other material in violation of copyright.

But again, it is as foolish and shortsighted to think of BitTorrent as being primarily about “steeling movies” as it is to think of http as a protocol for moving pornography. As the uses of BitTorrent for legal (even desirable) traffic become more widely known and increase, we see the expected result — greater use of BitTorrent and a continuing increase in the ratio of legal to illegal content sent via BitTorrent. Indeed, even mainstream content producers are increasing turning to BitTorrent for distribution, as demonstrated by this recent deal with Brightcove.

Comcast’s efforts to degrade BitTorrent and therefore decrease overall BitTorrent use thus create serious problems for itself and others. First, BitTorrent is not the cause of Comcast’s problem. Comcast’s “problem” is that it has users that want to move large files. That’s why they subscribe to big, always on, broadband pipes. People who do low-bandwidth stuff stick with dial up. And, as the use of the King James Bible for the test object demonstrates, Comcast’s actions target the use of the application (BitTorrent) rather than whether the underlying content is legal or whether the file is large or small. It is a rather crude (but cheap and privacy friendly) way of reducing overall traffic by teaching users to distrust particular applications.

In the short term, therefore, Comcast will likely see an overall drop in traffic on its systems as users stop using BitTorrent — although with a rise in customer dissatisfaction. After all, this method proved enormously effective for Microsoft. All it needed to do was render competing applications sufficiently unreliable that people stopped trying to use them because the available Microsoft application worked better with Windows than the competitor. Similarly, because Microsoft had sufficient market share, application designers would seek to satisfy Microsoft’s demands at the expense of other operating systems. Indeed, many developers simply stopped developing applications for non-Windows systems, because it didn’t pay well enough. So I have no doubt that Comcast can achieve its desired goal (reducing BitTorrent use) by degrading reliability for random users in a random fashion, as illustrated here. And, as demonstrated by Microsoft, it doesn’t matter how much your customers hate you as long as they keep buying your stuff — at least in the short term.

But users still want to move large files, and an increasing number of content producers want to sell them large files. Trying to move them in a traditional way — the next logical step — will increase traffic both for Comcast and for everyone else. Because one of the advantages of BitTorrent is to distribute the traffic load more broadly across the entire network. By forcing people to use less efficient means to move large files (and impact on Comcast subscribers — which make up 12% of the U.S. broadband market — impacts every single vendor of large files and therefore every network that carries this traffic), Comcast increases overall network congestion as a whole. Because people trying to move large files through less efficient means cross multiple networks, and these networks must now carry this larger traffic load. We call this a “negative externality.” In real world terms, it’s like Comcast dealing with toxic waste by dumping into the local river and letting everyone else downstream handle it.

But Comcast — if permitted to engage in this kind of conduct — has done more than degrade trust in BitTorrent. As it and other providers engage in such tactics, it effectively destroys the ability of anyone to provide a viable edge-based means of reducing traffic flow. Because, again as I described in my original post on the subject, who the heck is going to make an edge based application that dies as soon as it hits Comcast’s (and other provider) networks? It’s not worth the investment, because you can’t count on getting customers. Would Brightcove have paid if it knew that Comcast customers would likely stop using BitTorrent because Comcast rendered BitTorrent unreliable? Maybe, but I can guarantee they would have paid a lower price.

So just as applications designers gradually gave up trying to develop systems Windows degraded, I expected to see a decline in efforts to generate edge-based (rather than core-based) means of making traffic more efficient and decreasing overall congestion. That leaves us exclusively with core-based traffic management rather than edge-based. Worse, it means either that the carriers with the bulk of broadband subscribers will set a single standard for the majority of the U.S. internet market — the same way Microsoft became the de facto standard operating system for desktop application developers, or we will see a mutually exclusive standards war between the largest network, each trying to capture the edge market by imposing its standard as the means to reach its subscribers.

Oh goody! A choice between a cartel exacting monopoly rents and imposing a single solution on an industry, or a massive standards war balkanizing the ‘net with all the inefficiencies and costs for edge-based providers –particularly providers of high-bandwidth “Web 2.0” user generated content. It’s like a public policy dream come true, assuming of course you are some other country dreaming of how America can lose its premier place in the new economy (which, thanks to crappy broadband policies that permit this sort of thing, we losing quite steadily. For us, it sucks rocks again (although on this one we have a choice of which types of rocks to suck).

So, in the short term, Comcast’s efforts to control its “quality of service” merely make life miserable for its users. Longer term, it also shifts the problem of moving large files around to every other network, trashes a legal delivery system that content providers have started to embrace, boomerangs back on Comcast, and sets the stage for either an extraction of monopoly rents or a standards war that imposes massive costs and inefficiencies on the internet economy as a whole. And that’s accepting Comcast at its word that it is merely trying to manage network traffic (without engaging in those pesky and expensive network upgrades) and that it is not (yet) trying to force users to stop using BitTorrent so they will buy video programming through its rival Video on Demand (VoD) system.

Why Large Content Providers Should Get A Clue, But Won’t. As we can see from the description above, the content providers have a fair amount to lose here. Comcast has just demonstrated that it can render their investment in delivery systems utterly worthless. They should expect that the next move for Comcast and other providers would be to start charging them through the nose for premium delivery. Not only would that drive up cost for content providers, it will shrink the online “pie” overall. Online consumers are rather price sensitive, and where they pay more they expect increased value. Raise the price for online content by allowing ISPs to charge for premium delivery, or make delivery too time consuming, and subscribers either stop buying or start flying the Jolly Roger.

Video and music companies have seen what happens when a single entity or cartel gets control. Just ask broadcasters or independent programmers whether they get a fair market price from Comcast and Time Warner cable for their video programming. Ask the RIAA what it thinks of iTunes, which achieved dominance because the music companies forced Apple to adopt DRM, and the same DRM locks users into iTunes. The RIAA actually supported Future of Music Coalition in its campaign to wipe out payolla because a universe where Clear Channel controlled a sizable chunk of the market, while it squeezed out independent competition quite nicely, ended up costing the major labels big bucks to get air time, concert promotions, and the other necessary ingredients of delivering a product to the market.

So you would think a smart video or music label, aware of this history and seeing how Comcast just reduced the value of Brightcove’s investment in a distribution system it chose and can influence, would want to mandate net neutrality to keep control at the edge rather than allow the handful of large broadband access ISPs at the core to dictate terms or totally disrupt the market with a standards war. Alas, as best as I can determine, there are no smart video or music labels, at least among the big boys. Independents are another story, which is why they have favored net neutrality since this issue started to roll.

The Viacoms of the world, however, labor under the delusion that they can leverage the bottleneck control of the broadband access ISPs to stop piracy and kill competition for independents. After all, vertical integration in the form of ownership of distribution chains like Tower Records, as well as contracts effectively excluding independents from box stores such as Wal*Mart, proved extremely potent weapons against independents in the early and mid-1990s. Why embrace new ways of doing business, think folks like Sumner Redstone, when we can make deals to revive the old business models and tame the internet into something that lets us maximize our revenue by excluding competitors? And if that means cutting the Comcasts and AT&Ts of the world in for a taste, well, there’s plenty to go around — especially after we squeeze out the competition and can get back to our old price-fixing ways.

What these guys never get is that, once an entity owns the distribution pipe, it owns you. Yes, the cable companies and the telcos will cut you sweetheart deals today when they need allies. Why else would AT&T suddenly propose filtering for copyrighted material to the applause of the folks like Jeff Zucker at NBC Universal. But once the AT&T’s, the Time Warners, the Comcasts and Verizons have established a network architecture that gives them the kind of control that allows the, to filter for copyright and drive up the cost to rivals, what stops them from putitng the squeeze on you, as they have every time in the past you’ve been dumb enough to fall for this trick? Because letting these guys control content distribution by allowing them to degrade any edge-based delivery system like BitTorrent, even if it helps you reduce piracy in the short term, is basically putting your soft little scrotum in Comcast’s outstretched hand. And as you all should know from the last time you did retransmission negotiations, that hand will absolutely squeeze your naughty bits in an iron fist until it has wrung out your last dollar.

So unless you are into that sort of thing (and I make no value judgments here), y’all might want to see this AP test as a wake up call and reconsider whether you really want Comcast or other broadband providers to dictate to you what kind of distribution system you use. I know you think BitTorrent and other peer-to-peer systems are Satan Spawn, but the enemy of your enemy is not your friend here. If Comcast can disrupt BitTorrent, it can disrupt any other edge-based distribution system you come up with for yourselves. Is that the world you really want to live in?

Final Thoughts

As usual, I expect the opponents of network neutrality to defend Comcast’s actions. Scott Cleland, for example, considers it sufficient to say that Comcast’s actions are completely legal under the FCC policy permitting reasonable network management. While I don’t take the legality of Comcast’s actions here as a given (either under various laws outlawing deceptive consumer practices or under the FCC’s “network management” exception), it is also a red herring. For public policy purposes, the question is whether we want a world where Comcast (or other ISPs) can create this kind of interference with customer traffic while keeping it secret from the customers themselves. If common sense were not enough to provide us with the answer “Hell no!”, I would hope the above analysis would show why allowing Comcast (and other broadband ISPs) to have the freedom to mess with subscriber traffic is a really bad idea.

Stay tuned . . . .