Broadband and Burgernomics: Is the Monthly “Netflix Rankings” The New Big Mac Index?

Some years ago, The Economist developed an informal metric for determining whether currencies were overvalued or undervalued based on purchasing power. They looked at the price of a McDonald’s Big Mac in the target country currency as compared to the price in the U.S. This “Big Mac Index” works on a theory that currencies should converge on the cost of a standardized basket of goods, and Big Macs are an internationally standardized product using pretty standard food goods available locally, which makes them the ideal metric for comparison over time. (Let us set aside for the moment various tweaks for low wages.)

 

As explained by The Economist:”Burgernomics was never intended as a precise gauge of currency misalignment, merely a tool to make exchange-rate theory more digestible. Yet the Big Mac index has become a global standard, included in several economic textbooks and the subject of at least 20 academic studies.”

 

I thought of this when reading this piece by Jon Brodkin on the decline over time of several major broadband providers in the monthly Netflix Rankings. As reported extremely well and in great depth by Stacey Higginbotham at GigaOm (see here, here and here) and others (see, for example, this long and good piece by Jon Brodkin), a lot of factors go into how well or how poorly your ISP performs — particularly on high-bandwidth applications like video. With no standards by which to judge performance and with variability a factor over time, how can the consumer tell if its really worth it to shell out big bucks for top-tier speeds on your ISP when the video you want may still behave unreliably?

 

Then it occurred to me that the monthly ranking by Netflix of how well its service works on the major ISPs may work as the new Broadband Burgernomics. As I explain below, performance over time allows people to compare relative ISP performance (both the ISP’s performance over time and its comparative performance to competitors – if you are lucky enough to live in a market with good competitors). As with currencies, make a rough estimate as to whether or not the ISP’s speed and performance claims are ‘overvalued’ (i.e., the speed tier purchased does not deliver to the consumer a consistent experience the consumer would expect for the stated speed), accurate, or ‘undervalued’ (i.e., the speed tier purchased delivers to the consumer an experience better than what the consumer would expect based on the stated speed.)

 

I want to stress this doesn’t tell us whether the ISP is doing anything illegal or immoral. It doesn’t tell us anything about blocking or throttling. It also doesn’t tell us whose “fault” it is that performance drops — at least not for month-to-month variations.  But, as I explain below, using the same logic as “Burgernomics,” the “Netflix Rankings” index, taken over time, should provide a reasonable rough indicator for consumers as to whether or not their ISP can make that higher speed tier worth the money for the upgrade, and a possible indicator to policymakers and investors about deeper changes in the market.

 

Before all you carriers and the usual suspects who go apoplectic on the subject of ISP performance metrics and the various policy issues freak out, read my full reasoning below . . .

 

First, you might ask why we need anything other than the FCC’s official speed study (which you can use here), also known as the “Samknows Study”? Wasn’t the whole point of the speed study to provide a metric to resolve this question?

 

I happen to like the FCC report for what it does: provide a good, standardized metric that can serve as a starting point for discussion of broadband performance. I also respect the fact that when the FCC first pushed to develop its current speed study, it met fierce resistance from ISPs that did not want to see a standard study for a variety of reasons (some good, some less good) and faced real issues in trying to put together an acceptable way to measure the specific question of whether ISPs accurately advertised their speed.

 

All that said, as Stacey Higginbotham reported, the FCC Report does not fully capture user’s real time experiences. Critics of the FCC’s study note that providers have dedicated testing servers that avoid a lot of the factors that influence the user experience. ISPs respond that it is not fair to hold them accountable for factors beyond their control, and that to answer the question the FCC asked (“do ISPs deliver the speed they advertise”), you need to measure the network the ISP controls, not delivery of third party content here the third party makes many decisions that influence download speed.

 

As a result, we still have lots of different standards and arguments about what speed test accurately describes the user experience (and whether to blame the ISP, the content provider, or the complicated chain of third parties in the delivery system). Also, with a year (or longer) between FCC reports, a lot can change for consumers and the consumer experience.

 

 

The Theory Of Netflix Rankings And Burgernomics.

 

The average broadband subscriber, however, does not give a poop about these fascinating arguments or who to blame. The average user wants to know whether to shell out more bucks for a higher speed tier, and why high-bandwidth services they pay for like Netflix of Amazon Prime (which is why they pay the bigger bucks for higher speed in the first place) don’t work right. If consumers are lucky enough to have a choice of provider, they want to know which provider will actually deliver the services they want regardless of any kind of “blame.” Even if they don’t have a choice of provider, why upgrade to “Quantum Speed” if it doesn’t do me any good?

 

In thinking about it, I realized the problem is not unlike the currency valuation/ purchasing power question addressed by Burgernomics. A lot of factors may influence an exchange rate, many of them beyond the control of the country in question. But I don’t care about blame. I want to know whether the exchange rate actually reflects what I can expect to buy with my dollars (converted into local currency) or not.

 

Burgernomics works on the theory that the value of money is measured in purchasing power, and that the best way to compare purchasing power is the price of a standardized “basket of goods.” Big Macs are standardized across the globe. Not that every Big Mac is identical, as they have some regional variation depending on local taste and availability of goods. But they are standardized enough to be comparable.

 

We can make a similar argument for Netflix use across major ISPs. No, it’s not as precise. But we are not going for precision here. This is simply a useful rough estimate for consumers to compare ISP performance for typical heavy bandwidth functions.

 

For this to work, I am assuming the following:

 

1. Netflix content is not being uniquely singled out by any ISP. It is representative of all heavy bandwidth functions on the ISP.

 

1. Netflix use will be approximately equivalent on each ISP as a percentage of use by the subscriber population. It is a highly popular and widely distributed service, and even though the individual user may vary his or her habits, the sample size is large enough to cancel these out in the aggregate. As far as I know, nothing suggests that Cablevision subscribers are significantly different in their Netflix use habits than Charter subscribers or AT&T subscribers.

 

This last is important because we are dealing with averages over time. Any particular user may have wildly varying demands from other users or even in month-to-month usage. But over millions of subscribers, these differences average out. If the problem is that demand for Netflix spikes in the evening when folks get home from work, this should spike on all ISPs in the area at the same time. If Netflix demand spikes when the next season of House of Cards is released, then it will spike for all providers in proportion to their Netflix subscribers.

 

I don’t claim this is enough precision for a scientific study, but it makes a good rough estimate along the same idea as the Big Mac Index.

 

 

1. Netflix does not target any specific ISP, and has the same incentives as any other unaffiliated edge provider of high bandwidth content.

 

This assumption cancels out the problem of “bad decisions unique to Netflix.” Netflix can certainly make bad decisions over time, like anyone else. But if the problem is with Netflix, the relative rankings will remain the same. If the problem is, for example, Netflix declines to adequately provision itself for the volume of demand, that should impact all ISPs equally so that their relative rankings remain unchanged.

 

To the extent a problem outside the control of the ISP impacts a specific ISP, Netflix has the same incentive to fix it that any other provider of big content has. So if some change in something impacts a specific ISP (like a peering fight between Level 3 and Comcast), it is likely impacting other providers of high-bandwidth content (e.g., all of Level 3’s other customers) and Netflix has the same incentive to correct it over time to the extent it can (e.g., multihoming through multiple middle-mile providers).

 

 

Why This Makes Netflix Rankings The Digital Big Mac Index.

 

As with the Big Mac Index, the idea is neither to figure out why something is happening (although people may care about such things) or to strive for utter precision. It is a rough question over time. The Big Mac Index works, to the extent it does, because it serves as a useful, standardized proxy for general purchasing power over time.

 

For the same reason, the Netflix Rankings work as a suitable proxy for the relative ability of an ISP to handle high-bandwidth services generally. Remember, this isn’t about Netflix and any specific ISP any more than this is about McDonalds and Romania. Rather, for the reasons stated above, the Netflix Rankings tell me from a consumer perspective whether some ISPs are having issues, improving their performance, or if there is an overall systemic problem with the delivery of high-bandwidth services.

 

Nor does this tell me where the problem is (assuming there is a problem). It just provides me with a useful way to see if there is significant variation in how specific ISPs (or all ISPs) handle high-bandwidth services. If, from a policy perspective, I decide that I care (because the ability to handle high-bandwidth services is important to our economy, for example, rather than because I care about Netflix per se), this serves as a good, rough red flag that I might want to look at something.

 

 

Application: How Would This Work In Reality?

So lets go back to the recent news that three ISPs, Comcast, TWC, and Verizon FIOS, have shown slow but steady decline over in their Netflix Rankings over the last few months. What would this tell me as a consumer? What would it tell me as a policymaker.

 

As a consumer, the fact that this has been a slow and steady decline over time, rather than a one-time blip, indicates that these three providers are less likely (compared to others) to deliver the performance I expect for high-bandwidth content. The fact that it is impacting these three rather than all ISPs indicates to me that if I can switch to one of the competitors that is not showing a steady decline in ratings, I will probably have a better experience with high-bandwidth data. This doesn’t tell me whose “fault” it is, but I don’t care. If I can choose between Cablevision and FIOS, I say “well, Cablevision is delivering the goods, while FIOS seems to be getting worse at delivering the goods. Sucks to be FIOS, I’m heading for Cablevision!” That is how competition works in the two-sided market.

 

This comparison is useful even if, like me, you are not a Netflix subscriber. I don’t use Netflix, but I use a lot of Youtube and streaming video for my wonk habit. My son plays lots of graphic-intense games online. My wife is an IT pharmacist and works with high-bandwidth medical files. Since my underlying assumption is that FIOS does not target Netflix, I assume Netflix is a good proxy for these other, high-bandwidth uses. Netflix has the same incentive as Youtube to serve FIOS customers, so whatever is degrading FIOS’ Netflix Rankings is likely to degrade their Youtube or other high-bandwidth uses. So if Cablevision can handle Netflix better than FIOS, it can almost certainly handle Youtube and the other applications better.

 

But let’s assume I live in Montgomery County, MD, where Cablevision is not available. Let’s also pretend RCN is not available. My competitive choice is Comcast – which is experiencing a similar decline in Netflix Rankings over time. So switching from FIOS to Comcast doesn’t help me because whatever is impacting their rankings seems to be impacting them about the same relative to each other. Bummer for me.

 

But the comparison is still useful. FIOS keeps wanting to sell me higher speed. But is it worth it? I look to the Netflix Rankings and I see that FIOS is handling Netflix content less and less well. So is upgrading to a higher speed worth it? From this ranking, it appears that Verizon FIOS’ broadband is “overpriced,” i.e., it doesn’t handle the high bandwidth applications I want as well as I want.

 

If I were looking at this as a policy maker rather than a customer, the fact that some ISPs seem to be experiencing steady decline over time may make me ask why. Is it a systemic problem? Does it indicate some shift in the market of which I should become familiar? Why these ISPs relative to other ISPs?

 

It doesn’t mean I will necessarily do anything – especially if this is the only data point. The Treasury Department does not set policy by the Big Mac Index. But it is interesting, and may point me to other useful data.

 

 

Conclusion

 

My point here is not to call for any particular action, or even offer ay explanation. Rather, like the Big Mac Index, I think it provides a useful rough tool to try to get at things that consumers and policy makers actually care about. As always, I am aware of my limitations as a lawyer and would be interested in hearing from folks with relevant specialties on why they think “Netflix Rankings” makes a good digital version of the Big Mac Index.

 

I also want to stress that this isn’t a proxy for figuring out what is actually going on in the real world any more than the Big Mac Index tells me why a currency is comparatively over-valued or under-valued compared to other currencies. But it looks to me, at least preliminarily, like this could be a useful tool for some things.

 

Stay tuned . . .