Broadband Benchmarks

The September 2nd Federal Communications Commission National Broadband Plan workshop featured a panel of academics, policy directors and telecom executives that came together to discuss the role of metrics and benchmarks for evaluating the various dimensions of broadband across geographic areas and across time. The Benchmarks considered included variables such as broadband deployment and adoption, affordability and prices, quality of broadband services and levels of competition.

Gregory Rosston, the Deputy Director of the Stanford Institute for Economic Policy Research, introduced his discussion paper that looks into what programs exist in different states and what is it that drives adoption. An important factor was transaction costs. He said we should plan to use the benchmarks to find out what is actually driving the adoption rates. The first policy answer we want to come up with is to asses where we stand, then what benefits would come from programs to accelerate our position, and third, what are costs of the changes and can we justify the costs.

International comparisons of broadband are good but can be misleading

• OECD - measures Internet penetration per capita. This does not account for households.
• Pew Survey shows 60% of household have broadband, but very few people pay attention to the margin of error.
• Another important question is, how far behind are we? Do comparisons with other countries make a difference, do we need to compare on household level or on a consumer level?

It is important to understand the benefits of broadband

• Two kids of benefits, private and public.
• Studies on the value of broadband generally do not look at externalities - it is an important area of research to justify big expenditures based on externalities. Also, is Internet a general purpose technology like energy which would makes it harder to measure.

Costs are an important aspect to consider

• He tried to propose reverse auctions to minimize cost of new subscriber added.
• If benefits outweigh costs, what is the most efficient way to pay for it?
• What factors will increase broadband at the least possible costs?

AT&T's Richard N. Clarke, said benchmarking means taking a lot of data and condensing it into a form that is easy to follow over time or geographic location. Benchmarks need to show the policy goals of the commission, and policy goals should be related to the US customer. Goals should be broad, stable, ensure the relevance of the benchmarks. After setting goals then we must set benchmarks.

The benchmarks tend to fall into three specific categories:

1. Is broadband available? - Seems simple but it depends on geography. Availability of zip code too vague, availability at street address too burdensome
2. How does it perform? - Difficult because difference uses have very different performance requirements. Speed is generally used but there are many other performance components. Performance over a single day can very greatly. An index of performance components was suggested but hard because users have different weights for different performance components. Another solution is let consumers measure for themselves and conduct a pole based on satisfaction.
3. What is the price? - How to measure depends on policy goal of benchmarks? Looking for best deal or worst deal? Also broadband is normally lumped together in triple play for example. What price is used, the stand alone or the triple play bundled price?

Any benchmark must realize customer preference changes over time. Index must be broad in scope to be relevant for a reasonably long time.

OneEconomy's Scott Berendt said Congress must create a Broadband Progress Board -chaired by FCC and advised by various government agencies including the Departments of Commerce, Education, Agriculture, Energy, and Urban Development, as well as various other public and private entities and associations. The Board should implement and monitor the National Broadband Plan through the guidance of set benchmarks. The Board would inform and shape policy directives, drive innovation and assure that broadband will be affordable and available for all.

Goals for 2013 or sooner:

• affordable broadband to all country
• national digital literacy venture
• ubiquity of online public purposes
• all government services should be online
• fully digitized national emergency network
• all public and affordable housing should be wired
• 100Mbps or greater for health, education and public safety

Tools and Methods to meet the goals

• FCC 477 forms
• Asset mapping
• consumer surveys
• civic participation
• online crowd sourcing

Public Knowledge Legal Director Harold Feld pointed out the differences between benchmarks and goals. Goals are what we want to come out of the broadband plan. Benchmarks is the stuff we have to measure the progress of the broadband plan so that we know that we are on track. Benchmarks have to be informed by the goals of the statute. The goals in the statute are very vague and complicated.

In order to benchmark properly we need to understand not just a broadband market but a broadband ecology. To do what the statue tells us to do we need to really understand how broadband truly affects us.

Three areas of focus - traditional last mile, middle mile, qualitative metrics covering vast spectrums of the quality of life.

Need to move away from traditional FCC approach of notice and comment we need allow for real time data that is dissected and shared with the public. We need consumers, crowd sourcing, and application that can do random measurement of speeds and how the network is behaving on a regular basis. We need to use technologies and methods that the private sector has been using for a long time.

Biggest advice is don't try to do this alone, coordinate with the other agencies and organizations. Benchmarks should also be re evaluated consistently to make sure they are informing our goals.

Catherine Sandoval, Assistant Professor of Law at Santa Clara University, said that in order to define broadband, we need to distinguish different types of access and restrictions to access created by different service providers, instead of focusing on speed.

• Speed does not measure whether one product is actually a substitute for another. Nature of the level of Internet access is very important.
• Applications and application use, attachment of devices, congestion policies all make a difference and affect speed.
• Due to the Brand X decision the number of restrictive practices has also increased and affects the level of broadband.

Need to report on gaps in Internet access. Need to make sure we measure all the gaps including surveys done in languages other than English.

• In terms of the rural gap, many federal rules exclude very rural areas near a major metropolitan city
• The Pew studies are also not done in Spanish and therefore they are missing out on entire Latino communities.
• Last big gap issue is computer ownership.

Jon Eisenberg, the Director of the Computer Science and Telecommunication Board at The National Academies, compared 1) a 2002 report that provides broad assessment of landscape and recommendations for speeding deployment and focuses on broadband definitions and 2) a second report is a 2009 report looks at information technology research and development ecosystem.

2002 Report

• Looks at a broad survey of broadband technology and policy, multiple dimensions of broadband, various dimensions of quality, service and upstream and downstream bandwidth.
• Raises question of whether connection is shared via a home network?
• Question of addressability?
• Controls on applications and content?
• What is delivered on and how?
• The survey asked about who benefits from workable definitions
• Consumers
• Service providers
• Applications and content developers
• Policy makers
• Public interest groups
• Two definitions of broadbandband
• 1 - local access link performance should not be the limiting factor on a users experience in running today's applications
• 2 - Broadband access should have enough performance and wide enough penetration of the performance to encourage the development of new applications,
• Applications as performance indicators
• Is performance perceived as improving or deteriorating
• Are new application that depend on high bandwith emerging

2009 REPORT

• Broadband Mobile - Key objectives
• Ensure infrastructure to lead the world
• US has been loosing ground
• Should establish targets to regain leads
• What it means to be world class? Reaching ambitious targets.

QUESTIONS

Is price relevant in assessing whether broadband affordable? How can the FCC measure price in broadband given difference is types?

Harold Feld answered that we need to break it down to regional levels with reference to personal income. We have to find the gaps. There is a strong correlation between low income and adoption and therefore affordability is a factor in adoption. The best way to track price is through census blocks (which tracks income) and see whether advertised price is the same as real price. We could see if people are getting particular deals or not. Can people in a specific area actually afford to but the broadband?

How do we deal with the fact that price includes different offerings, different charges, promotional periods, early termination fees. What exactly are the services buying?

Sandoval answered by saying that we need to ask if the products are comparable? Look through many dimensions, given the restrictions if the price of one grows would you substitute it for another? Access to credit and access to bank accounts can be a factor. Garment workers in SF deal with pre paid cell phones because they do not need a bank statement in order to start up a plan for one.

Clark suggested that given variety of uses we should create a usage profile? Most people have only very basic uses. What is the relevance of the usage profile within the community?

Rosston - how you measure price depends on what your goal is? For assessing affordability then you need to look at census block and that information. Amortize cost of hook up charge based on an average cost for household. If we want a price index for competitive level and prices over all - should there be price index for different services?

Feld - there is no easy metric. Do not try to figure this out alone. Expect and encourage that others looking at problem over time will add and help. What is your definition of broadband will affect the price. Purpose of the statute is to drive to the maximum possible utility of the network.

With computing power do a survey with all price variable of every provider or each block in the US. We would have a big database, but would it be usable? It would need to reflect complexity but must show if progress has been made?

Feld - you will collect more info because you cannot do this alone. Academics would love it.

Since there are limitations of polling end users, what are suggestions on where are we going from here?

Clark - what uses do we want Internet to support, and create a profile of what performance is needed to support those uses.

Rosston - Price index asked what does it cost in 25 different cities at different speed for different times? This gives you a baseline where you can compare profiles of prices overtime.

Berendt - this is evolving process, figure out what bottom is, 5 years ago uses and application were very different than today. Compiling primary needs as a floor.

Feld -Perhaps we want to evaluate the over all utility and use to support the maximum uses.

Sandoval - focus on restrictions that create a fundamentally different type of product. The restrictions make different services. With respect to polls, time needs to be spent with the special minority groups.

Eisenberg - want to measure uptake of new more demanding applications

We should be making use of other entities to collect the data? Financially, will entities take on this task for long term? How do we make sure entities do not slant toward one provider or another?

Feld - many private sector companies make their living doing this? Neilson has been doing this for a while? They are sustainable because they break it down into something simple. There is a user effect but people are sophisticated about that. Develop applications that are standard reliable and interactive such as a tweet that will allow the user to stay up to date in reporting broadband usage, speeds etc.

Eisenberg - many applications do their own monitoring of the networks. Perhaps we could aggregate that data.

Has there ever been an attempt to create a useful index?

Feld - there has been work on a database of surveys to see what people have been asking and what indexes they have they been using?

Clark - give people different technologies and ask them how did it work?

Sandoval - user perceptions - sometime users don't appreciate what is going on. Sometimes latency of congestion may not be noticed. As much as deep packet is criticized, the reality is that many carriers are employing it.

Berendt - digital literacy and digital awareness also must be a part of this discussion. People need to be acquainted with what is out there and what is valuable?

Feld - We have been doing E-rate for 10 years and know how many people have applied and how many have collected. We have NO knowledge whether or not it has made a difference to educational outcomes? When examining whether meeting the goals, we need to ask why they are happening.

Long term goal of 100 Mbps in 100 million homes and small businesses by 2020 - how should the task force balance long term efforts with short term getting to targeted populations as soon as possible?

Eisenberg - Decide how much to invest, benchmark yourself against competitors no other good way to do it.

Availability - Should we give up on collecting the form 477 information?

Feld - Some are critical of form 477 because it is not helpful for this purpose - maybe helpful for the broadband map. If you filed info in real time it would be more timely and accurate.

Would it be useful to include a question about broadband use in the upcoming census?

Rosston - need questions on broadband in the census

Feld - need to make questions as simple as possible. We cannot ask do you have broadband. Questions should be oriented toward factual things that take quantifiable measurements. Example, have you bought something using an Internet connection.

Sandoval - could be useful on gather info on the lack of computer access. Census currently asks do you have telephone and what type? We should ask how are you accessing the Internet?