Netflix, Comcast and Our Changing Internet

Have three sentences ever generated so much ink? On February 23, 2014, Comcast released the following statement:

Comcast and Netflix announced a mutually beneficial interconnection agreement that will provide Comcast’s U.S. broadband customers with a high-quality Netflix video experience for years to come. Working collaboratively over many months, the companies have established a more direct connection between Netflix and Comcast, similar to other networks, that’s already delivering an even better user experience to consumers, while also allowing for future growth in Netflix traffic. Netflix receives no preferential network treatment under the multi-year agreement, terms of which are not being disclosed.

Peaches and cream, right? Two companies reach an agreement and the customers of both companies benefit. Nothing to see here.

Well, let's wade into some of the digital ink that’s been spilt since the announcement and see if this is a big deal or not.

The Wall Street Journal broke the story reporting that Netflix had agreed to pay Comcast to ensure Netflix movies and television shows stream smoothly to Comcast customers. In exchange for payment, Netflix is to get direct access to Comcast's broadband network.

Why would Netflix need to do that? Earlier this month we learned that Netflix’s video streaming performance on Verizon and Comcast networks has been dropping for the past three to four months. Verizon FiOS suffered just a tiny drop from 2.22Mbps to 2.2Mbps from October to November, but then it went down to 2.11Mbps in December and 1.82Mbps in January. Verizon DSL dropped for four straight months from 1.42Mbps to 0.97Mbps, with the largest single-month drop occurring between December and January. Comcast also dropped for four straight months, from 2.11Mbps in September to 1.51Mbps in January.

Verizon and Comcast have each been involved in disputes with video streaming services like Netflix and YouTube. When these negotiations fail, users suffer. In other words, bad video performance is often caused not just by technology problems but also by business decisions made by the many companies involved in delivering Internet traffic from content providers to end users.

These disputes are about how large networks exchange traffic. “Peering” is voluntary interconnection of administratively separate Internet networks. In the traditional architecture of the Internet, an “edge provider” (a Netflix or Yahoo, say) paid to connect to an Internet backbone provider to move the edge provider’s content toward an end user. Along the way, the backbone provider might exchange traffic with another backbone provider before the content/traffic is delivered to a residential Internet service provider (ISPs like AT&T, Comcast, Time Warner Cable, Verizon, etc) and, finally, to the end user. Although the edge provider paid a backbone provider, the end user paid the residential ISP and the residential ISP paid another backbone provider, the two (or more) backbone providers involved in delivering the traffic did not pay each other. Their peering agreement said, basically, ‘You take my traffic, I’ll take yours and we’re good.’

How the Internet Has Looked:

[Source: Washington Post. If you are having problems viewing, see: http://benton.org/node/175498]

Peering agreements are implemented by physical infrastructure that literally connects to the backbone providers. But business disputes can prevent peering infrastructure from being upgraded which can lead to too much traffic needing to fit through too small a connection. [Think not enough lanes on the George Washington Bridge.]  This affects all Internet traffic being exchanged by the backbone providers, but it places a more noticeable toll on streaming video because of how much bandwidth it requires.(1)

Earlier this month, ars technica reported that the network connections between Cogent and Verizon, crucial for carrying streaming video and other content to Verizon's home Internet customers, "are full," according to Cogent Communications CEO Dave Schaeffer. "They are more than full. They are so full that today a significant amount of packets are being dropped between the networks." Cogent, by the way, is a backbone provider for Netflix and other companies. Verizon wants to ditch the free peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix. Cogent has refused to pay. As negotiations stall, Netflix performance has dropped measurably for months on both Verizon and Comcast. Cogent claims this is because Verizon and others -- but especially Verizon -- are refusing to upgrade the connections between networks. Cogent points out that Verizon offers its own streaming video services, such as Redbox Instant, and thus has an incentive to harm Netflix traffic.

There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. They can do this by adding ports, adding capacity to ports, or peering in new locations. With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Cogent’s Schaeffer. Comcast and Time Warner Cable are demanding payments similar to the ones Verizon wants, and Cogent's connections with those ISPs are reaching capacity, too.

And residential ISP consumers can’t buy their way out of the problem. Even if you subscribed to a faster Internet connection, the Internet traffic to and from you is still being slowed because of these peering disputes and the bottlenecks they create.

The Netflix-Verizon Dispute:

[Source: Washington Post. If you are having problems viewing, see: http://benton.org/node/175498]

"We bill our customers and we connect them to the Internet, and it's our job to deliver to them the best quality possible,” said Schaeffer.  “Verizon is billing its customers, and it's got a lot more of them and it bills a hell of a lot more, and they're not committed to delivering their customers what they promise the customers they're going to deliver."

Why is that? Well, the Internet is changing. For example, in 2006, Verizon acquired the formerly independent backbone provider MCI, helping to turn Verizon into a major backbone provider in its own right. Now, Verizon plays the role of both backbone provider and residential ISP. That puts Verizon in a much stronger negotiating position, because Cogent doesn't have any practical way to route around Verizon. If Cogent wants to reach Verizon's customers, it needs to cut a deal with Verizon.

Dan Rayburn, an executive vice president at StreamingMedia.com and principal analyst at Frost & Sullivan, offered a long take on what he sees is happening in the Comcast-Netflix deal. He notes that each ISP has a peering policy (here’s Comcast’s). ISPs like Comcast will allow transit providers like Cogent to connect to their network, for free, in what’s called settlement-free peering. However, once the transit provider sends more traffic to the ISP then they are allowed to, per the ISPs peering policy, the transit provider pays the ISP for more capacity to get additional traffic into their network. Remember, Netflix is the one paying Cogent and Cogent is selling Netflix on the principle that it can get all of Netflix’s traffic into an ISP like Comcast. As a result, Rayburn argues, Cogent has to take all the necessary business steps to make sure Cogent has enough capacity to pass Netflix’s traffic on from Cogent’s network to Comcast. But Cogent isn’t doing that.

Rayburn pins the poor quality streaming on Cogent’s refusal to pay Comcast to add more capacity, even though Cogent is taking Netflix’s money for the service. Cogent is charging Netflix for a service it can’t deliver. Some are arguing that Comcast should be the one to pay to upgrade that connection with Cogent since Comcast charges consumers to get access to the Internet and is double dipping by charging both the consumer and the content owner. In reality, Rayburn argues, Comcast isn’t. Netflix does not need to go direct to Comcast and pay them anything. Netflix chose to because the company could not get the level of service it was paying Cogent for directly. Netflix has also decided it makes more business sense for the company to build its own content delivery network (CDN) instead of relying 100% on third party CDNs (companies like Akamai, Level 3 and Limelight) like it used to. Netflix service was fine when it used these CDNs, Rayburn notes, because those CDNs already have their servers connected to ISPs like Comcast and have put in place all the necessary links, both free and paid, to guarantee, via an service-level agreement (SLA), that they can deliver Netflix’s video. A SLA is a contract between a network service provider and a customer that specifies, usually in measurable terms, what services the network service provider will furnish.

What the Netflix-Comcast Deal Looks Like:

Rayburn points out that Netflix has multiple options in the market for delivering good quality video, but Netflix chose to build their own CDN and change its delivery strategy because the company want to have more control over the network and save money. Netflix gets a guaranteed level of service from Comcast, but Netflix does not get any prioritization in the last mile. 

With Netflix paying to connect directly to Comcast, Cogent (and other Netflix vendors like Level 3, Tata, XO, Telia, and NTT) is likely to be cut out as the middleman. And Cogent may lose its battle with Verizon, too. In the long run, this all could mean the elimination of independent backbone companies. So what the Internet ecosystem might win in efficiency, it might lose in competitiveness as residential ISPs gain more and more market power. Recall, too, that in the background of this deal is Comcast's purchase of Time Warner Cable, another big ISP. If that deal is completed, Comcast will control roughly 40% of households subscribing to high-speed broadband access.

On February 26, Consumers Union sent letters to the Federal Communications Commission and the Department of Justice urging them to take a hard look at the Comcast-Netflix deal. The letters asked the agencies to  investigate whether Comcast is abiding by net neutrality obligations it agreed to when it purchased NBC Univiversal.

“We believe it raises serious concerns about the clout of Comcast, its ability to affect the prices and quality of service it offers consumers, and the alarming precedent it sets for the entire marketplace,” said Delara Derakhshani, policy counsel for Consumers Union.

With the Comcast-Time Warner deal facing review from both the FCC and the Department of Justice in the coming months, expect the Comcast-Netflix deal to continue to get lots of attention, too. 

Notes:

1. These peering disruptions have bigger implications than just streaming (entertainment) video. As voice over Internet Protocol (VoIP) becomes the norm for telephone service, that service, too, could be degraded. “If you've got VoIP and you're trying to run Skype, it may not work because the pipe was full, with your neighbor down the street trying to watch Netflix," Schaeffer said. "It's not only the video user. Every Internet user is suffering today in their ability to access all the applications, content, and other users across the Internet."