The FCC released a fairly thorough report on the widespread 9-1-1 failure that followed the June 2012 “derecho” windstorm. For those who don’t remember, the derecho differs from most weather events by coming up almost without warning. According to the report, carriers had approximately two hours of warning from the time the derecho started in the Ohio Valley to when it hit the D.C. Metro region.
As a consequence of the damage done by the derecho, Northern Virginia experienced a massive failure of its 9-1-1 network, leaving over 1 million people with working phones (at least in some places) but no access to 9-1-1. West Virginia experienced systemic problems as well, as a did a scattering of locations in other states impacted by the derecho. Verizon maintains the network in Northern Virginia, while West Virginia is managed by Frontier.
In both cases, the report concluded that both Verizon and Frontier failed to follow industry best practices or their own internal procedures. To be clear, this was not a massive dereliction of duty. But the accumulation of some corner cutting over here, some poor practice over there, meant that when the unpredicted crisis hit the system suffered critical failures precisely when most needed. Unlike just about every other part of the network, where providers balance the cost of hardening a network against potential events with a number of other factors, the core 9-1-1 system is explicitly supposed to remain operational in even the most extraordinary circumstances. It is the foundation of public access to emergency services. As long as I can contact the phone network, I should be able to get 9-1-1 service. Public safety responders rely on the public reporting emergencies so that they can efficiently deploy resources as much as the public depends on its ability to contact emergency services through 9-1-1.
It is particularly shocking to see this happen with Verizon. As I’ve observed before, Verizon generally prides itself on maintaining a high-quality network. Customer satisfaction surveys generally agree with this self-assessment. It is part of Verizon’s overall business strategy: build the best network, attract the customers willing to pay a premium for it. Yes, part of that strategy means selling off or generally neglecting the copper footprint outside the FIOS territories. But the 9-1-1 failure occurred in Northern Virginia, precisely the kind of high rate-of-return territory Verizon targets for top-of-the-line service. So what happened?
What happened in the derecho illustrates the problem of relying on general incentives and industry best practices instead of regulation for critical safety and reliability issues. In understanding this lesson, we need to understand that it is not that Verizon (or Frontier, or other companies) are evil or greedy or callous or any of the other delightful adjectives people like to use to personalize this. It’s not. Verizon and Frontier understand about keeping 9-1-1 going just fine. They are all for it. They are part of the group that develops the industry best practices, which they supplement with their own internal procedures.
Nevertheless, in economic terms, maintaining 9-1-1 is a deadweight cost of doing business. Expenditures on 9-1-1 do not contribute to the bottom line in the way fiber deployment or spectrum acquisitions do. Maintaining the 9-1-1 system is essentially a “public safety tax.” Yes, they get to recover the expenses in a general way through fees tacked on to the phone bill. But when deciding whether to spend a dollar more on 9-1-1 or how carefully to follow industry best practices rather than cut corners, the rational network operator has plenty of countervailing incentive at the margin to divert that dollar from 9-1-1 to something that contributes to the bottom line, or to push off some recommended maintenance just a little bit, or otherwise cut a few corners here and there.
The other important lesson is that, when dealing with emergency services in extreme conditions, it takes relatively few errors to have potentially far reaching impact. This makes 9-1-1 even more expensive to maintain. What can pass as a reasonable practice in managing a giant enterprise like Verizon’s telephone system is unacceptable in 9-1-1. Since all humans make errors from time to time, this means additional checks and redundancies (and workers). Again, while carriers have plenty of generic incentive to make sure their 9-1-1 systems remain intact, but for any specific decision about investing resources, the line worker or manager will invariably weigh the cost against much more attractive alternatives.
Regulation Comes In Where Industry Standards Are Not Enough
Regulation and enforcement change the incentive equation at the margin. First, in a behavioral way, making a rule mandatory and putting in an enforcement and monitoring structure generally make people take the requirements more seriously. To take an every day example, we all have incentive to drive carefully. We still have posted speed limits, because that sets an expectation about what it actually means to drive carefully. We have enforcement and penalties, because otherwise people decide to let their own judgment about what’s safe be their guide.
The other rather direct way it alters incentive is that it increases the direct cost of not making the investment. To use the speed limits analogy, I may still decide to go 50 in a 30 MPH zone because I’m in a rush and decide to risk it. But I’m a lot less likely to do it in a photo-enforcement zone and where the fine is doubled for going more than 15 miles above the speed limit.
In looking at the lessons from the derecho, we need to recognize that “industry has incentive to do X” is often not enough. In particular, in the case of emergency and reliability requirements for networks, we need to recognize that the general incentive of carriers to make sure their network stays up in a crisis doesn’t guarantee the industry will do what they need to do, because industry participants also have incentive to spend that money on things that more directly contribute to the bottom line.
That has several immediate implications. First, the FCC needs to make sure that all carriers are up to standard on 9-1-1, especially those carriers that actually maintain the 9-1-1 network rather than simply route 9-1-1 calls to the network. As I said above, Verizon and Frontier are not uniquely bad actors. Odds are good that every carrier — while generally compliant and taking their 9-1-1 obligations seriously, has similarly cut a few corners or neglected a few procedures here and there. Why? Because they all have the same incentives. A few spot inspections to make sure every carrier is on top of things will help prevent any similar 9-1-1 failures elsewhere.
Longer term, as the FCC considers both the response to Hurricane Sandy and the future of the phone network, we need to carefully consider where we need to rely on regulation rather than on industry incentives and self-regulation. That certainly applies to 9-1-1, but it also applies to network reliability generally. No network wants off time for any reason. But that does not gaurantee that carriers will find the right trade off between cost and benefit that we as a nation need in order to have a reliable infrastructure. When our broadband networks go down, it is more than an inconvenience. It disrupts business and disrupts people’s lives. Carriers have incentive to minimize dowtime, but they have other incentives as well. We need federal and state agencies to be willing to push carriers to spend that extra dollar when it’s needed, even if they would rather spend it on something else.
In short, Commissioner Rosenworcel is spot on in her statement when she says “it is time for an honest accounting of the resiliency of our Nation’s network infrastructure in the digital and wireless age.” That starts with recognizing that actual, enforceable rules with real teeth are necessary to make sure that core emergency functions work properly. General incentives are all well and good, but for critical emergency functions, they are simply not good enough.
Interesting read on the report. I would like to point out a few things —
1) Not everything particularly revolves around best practices. Some are regulations getting in the way. For example generators appear prominently in the report. Why? Because generators are the last thing that a Telco likes to deal with. Why? Because the regulations, EPA by the way, are so convoluted that even doing service on them triggers a ream of EPA filings. Diesel fuel is considered a dangerous substance by that organization. It is one of the reasons that service intervals and tests are conducted monthly at best. Maybe it is time for the EPA to review its own procedures.
2) Would you be willing to back the Telcos on tax provisions that would foster their abandoning the wireline infrastructure? To be quite honest it needs to be eliminated. Much of it is outdated and the revenue streams are no longer there to support the physical plant. (breaks my heart, I was a wireline guy. But time and tech marches on.) Celllular technology rules the day and we citizens and regulators need to understand that, that is the way it is.
Lets stop beating the dead horse and send him to the packing plant where it belongs. Otherwise we are just regulating the past.
As for the first, certainly the things that raise expense for prep ought to be considered. But these go into the rulemaking/standard setting process.
As for the second, I agree the focus needs to be on the upgrade for the PSTN. I don’t think we need tax breaks at this point. To the contrary, carriers are highly motivated to make the upgrade to save money. This is why the FCC’s Technical Advisory Committee (TAC) estimated we would be about 90% there by 2018. We will probably need to migrate the last ten percent of carriers who are small and can’t otherwise afford it, but we have time to figure that out.