Measuring broadband is an ongoing challenge for policymakers and, for many participants in broadband policy debates, often a source of frustration. The frustration about broadband measurement emanates from what seems knowable – at least it is about other infrastructure. We know where our roads and highways run. Today it is easy to know when they are clogged, where there are tolls, and how much those tolls cost. Electric infrastructure is essentially ubiquitous and it isn’t hard, in most places, to find out the cost of a kilowatt hour and compare prices among providers.

Broadband is another story. For at least a decade, policymakers and other stakeholders have sought to better understand broadband – where networks run, how fast they are, how much service costs, and who uses it. Answers on those issues turn out neither to be simple nor agreed upon. How many Americans do not have access to network download speeds of 25 Megabits per second (Mbps) – the Federal Communications Commission’s definition of broadband? It could be 21 million or it could be 162 million. When you measure network speeds, what do you find is the average speed in the United States? It could be 72 Mbps, or possibly 119 Mbps, or maybe something else. How many people have broadband internet subscriptions at home? It could be more than 80% or it might be something like 70%.

What follows is an overview of broadband’s measurement challenges and what they mean for policymakers and other stakeholders.

Network Deployment

Where networks are deployed is among the most hotly contested issues in broadband policy for two reasons. First, elected officials hear from constituents when they have either no broadband where they live or broadband service of insufficient quality. This leads to the second reason – policymakers wanting to close network deployment gaps with public funds, but wanting to know how to direct those funds to the places with the greatest need. Broadband maps are the tools to address this, but they can prove imprecise.

Let’s start with the Federal Communication Commission’s Form 477. The FCC has required carriers to report advertised speeds by census block using Form 477. That approach overstates broadband coverage because an entire census block will show coverage even if a carrier can provide (not does provide) service in a small portion of it – even just one address. This inability to determine the nature of service availability at the household or street level has received widespread criticism from policymakers and consumer advocates.

In response to these criticisms, the FCC in August 2019 issued an order on its new approach for determining the location of fixed-broadband networks – called the Digital Opportunity Data Collection. Per the order, the FCC will require fixed providers to “submit broadband coverage polygons depicting the areas where they actually have broadband-capable networks and make fixed broadband service available to end-user locations." In seeking better data, the FCC hopes to improve how Universal Service Fund subsidies are allocated. The order does not do away with Form 477 data collection.

There are other proposed solutions for the FCC’s broadband mapping problems. USTelecom, a trade association, is working with the FCC on a pilot program to conduct better broadband mapping in a number of states. The USTelecom approach relies on geolocation and provider data to explore network availability at the census tract level (a census tract is roughly akin to a neighborhood, but can be geographically large in rural areas). Other approaches rely on shapefiles, which situate broadband networks in polygon shapes on maps and offer a more granular approach than census blocks. Something to watch in all approaches is how user-friendly these approaches may be for stakeholders wishing to pursue independent analysis of broadband maps and how much information from the maps would be available to the public. Neither approach deals with an element desired in broadband maps from some stakeholders – pricing data for broadband offerings in specific locations.

These proposals align with bills currently before Congress. Senator Shelley Moore Capito (R-WV) has introduced the Broadband Data Improvement Act of 2019 to require carriers to provide data using geospatial information systems (GIS) that can generate shapefiles. The other piece of legislation is Broadband Deployment and Technological Availability (DATA) Act that requires more granular data from wired, fixed wired, and satellite providers, as well as “strong parameters” for the availability of mobile broadband data. This bill has similarities with the US Telecom approach. The Broadband DATA Act has been approved by the Senate Commerce Committee.

Network Speeds

Knowing where networks are deployed is one problem, but understanding network speed is another. The FCC’s Form 447 allows carriers to report advertised speeds, but advertised speeds can and do diverge from actual speeds delivered to subscribers. By how much and to what degree depends on different measurement techniques for assessing networks. When the FCC asks carriers to report advertised speeds, the result is a range of speed tiers available in different geographies. For instance, the FCC’s latest Broadband Deployment Report finds that 93.5% of Americans have access to terrestrial broadband at download speeds of 25 Megabits per second (Mbps) and 58.8% of Americans have access to 250 Mbps download speeds.

But how well this approach captures reality on the ground is unclear. Many worry that relying on carrier-reporting is fundamentally flawed and speed tests (like those offered by Ookla and Speakeasy) are better ways to understand network speeds.

Speed tests, while useful ways to learn about network speeds, do not necessarily settle the issue. The FCC has its Measuring Broadband America reports, which use the company SamKnows to conduct tests on network speeds. The SamKnows approach places devices to measure performance at a point in consumers’ homes before the router directs data to people’s access devices. This means that intervening factors (e.g., a user’s home Wi-Fi network) do not influence the measurement result. Household selected to be part of the SamKnows study receive a Whitebox device to measure network speed, as well as other network attributes, such as latency and jitter.

The Measuring Broadband America reports document a steady increase in network speed from 2014 to 2017, with median speeds going from 31 Mbps in 2014 to 72 Mbps in 2017. The reports, however, are based on the findings from 12,000 households and do not represent a random sample of American homes. That is because SamKnows issues a call for volunteers for the program; out of the pool of volunteers, 12,000 are selected with an eye toward having enough participants from large internet service providers. With a limit of 12,000 in the pool of participants, the approach does not permit understanding network speeds in different geographies. The FCC’s speed reports do, however, compare advertised speeds to its measurement results. In its eighth report, the FCC finds that “most” internet service providers in the sample exceeded 100% of advertised speeds during peak hours.

Yet the findings from the Measuring Broadband America reports do not align with results from another well-known network measurement approach – crowd-sourcing. M-Lab allows anyone on the internet, with one click, to determine the speed of the network they are on at a given moment. Given enough participants in the speed test, it is possible to map results to particular geographies. A team of researchers at Penn State recently used M-Lab to test network speeds for the state, gathering 11 million test results in 2018. When the Penn State team mapped results at the county level, the results showed none of the state’s 67 counties had 50% of the population with access to download speeds at 25 Mbps. Speed gaps were greatest in rural areas. Other researchers have used M-Lab to measure speeds in particular areas, with one such undertaking in Iowa showing that internet users there experience 25 Mbps speeds 22% of the time, even though the entire state, according to the FCC, has 25 Mbps coverage.

M-Lab is not the only speed test available. Another example is Ookla’s Speedtest Global tool, which compares network speeds around the world. Its current index shows the United States ranked seventh in global fixed broadband rankings, with an average download speed of 119 Mbps, about twice the worldwide average. For inclusion in Ookla results, a test must come from a provider that has a significant number of customers in a geographic area; the provider must also meet a minimum threshold for tests from unique devices over a six month period. Note the gap between the FCC and Ookla on measuring network speeds in the United States – 79 Mbps compared with 119 Mbps.

How about another speed test? Akamai, a content delivery network, publishes the speed recorded when users seek content from firms who use Akamai to help optimize the content delivery. For the United States, Akamai’s 2017 first-quarter results show an average broadband speed of 18.7 Mbps.

If self-reported results and speed tests are not fully satisfying, why not peek into the network? A recent undertaking by Microsoft along these lines has made waves. Microsoft took advantage of the fact that so many household computers use its operating system and that the software is updated over broadband networks. By examining the time it took for system updates, Microsoft calculated broadband speeds people experience – and found that 162 million Americans do not have broadband speeds that meet the 25 Mbps download speed threshold. That’s a far cry from the 21 million lacking broadband estimated from the FCC’s Form 477 data.

Adoption

Measuring broadband adoption – who subscribes to broadband – is perhaps less contentious than network deployment, but not without nuances. The primary way stakeholders understand broadband adoption is through surveys of individuals or households. Let’s look at three surveys policymakers often rely upon.

American Community Survey

The Broadband Data Improvement Act required the Census Bureau to ask questions about home internet use in the American Community Survey (ACS). For the ACS, households receive notices through the mail that they have been selected for the survey, and they can respond through the mail, using the internet, or by telephone. If contacted households do not respond, ACS follows up with phone calls to ask that the survey be completed. In 2017, 93.7% of contacted households completed the ACS. The ACS started asking the question about home internet use and type of connection in 2013.

The large sample size of ACS allows analysis of fairly disaggregated geographic units. Since the ACS is an ongoing survey, Census aggregates the data in different ways. For analysis of census tracts, ACS aggregates data over five years. This means some 17.5 million households are available for analysis. The 2013-2017 5-year ACS estimates were released at the end of 2018. The ACS also releases 1-year estimates, which has a sample size of 3.5 million; these 1-year ACS estimates are appropriate for places with populations of 65,000 or more.

The trade-off in using the 1-year estimates versus 5-year has to do with geographic granularity and phenomena of interest to the analyst. The 5-year estimate allows investigation of a certain variable – say broadband subscriptions at home – in a specific neighborhood. As noted, this requires combining data over 5 years. However, broadband adoption may change over five years – and, in fact, did from 2013 to 2017. To understand the changes, the 1-year estimate is the tool to use, but can only paint a geographic picture for places whose population is 65,000 or more. In 2013, ACS shows that 73.4% of Americans subscribed to broadband at home, a figure that rose to 83.5% in 2017. The entire 2013-2017 5-year estimate, however, shows that 78.3% of Americans with broadband at home.

For understanding broadband adoption in a particular place, ACS data offers opportunities, but some limitations as well. For a sizable place (i.e., a population over 65,000), the 1-year estimate offers a snapshot for a given year and allows the analyst to do trend analysis on prior years. In other words, for cities with a population greater than 65,000, the 1-year estimates can provide a portrait of broadband adoption change from 2013 to 2017. For a neighborhood, the 5-year estimate is the appropriate tool, but might not be a fully accurate point estimate for broadband adoption in a neighborhood in 2017 (the end of the 5-year estimate window). And the 5-year estimate does not permit trend analysis for places under 65,000 in population.

Current Population Survey

Another way in which the federal government measures broadband adoption is through the Current Population Survey (CPS). Since the late 1990s, the National Telecommunications and Information Administration (NTIA) has collaborated with the U.S. Census Bureau for a Computer and Internet Use Supplement to the CPS. The CPS is conducted by telephone and has a sample size of about 60,000 households, which permits analysis of results on a state-by-state basis. The Census has been asking about people’s computer use since 1984 and internet use in 1997. Response rates are high for CPS supplements, with about 85% of households contacted completing CPS surveys. NTIA has created its Digital Nation Data Explorer to allow users to query the survey data from pulldown menus.

The CPS allows NTIA to ask not just about whether people have broadband at home or not, but also to explore specific topics about online use. This lets NTIA explore people’s attitudes about privacy, barriers to broadband adoption, use of the internet for job search, and more.

As to measures of broadband adoption, the ACS and CPS diverge somewhat. The ACS in 2017 found that 83.5% of Americans subscribed to broadband at home, while NTIA’s CPS found that 85.9% of Americans had broadband. (Given the large sample sizes in each survey, the 2.4 percentage point difference is statistically significant.) 

Pew Research Center

Since 2002, the Pew Research Center has been conducting national, random-digit-dial telephone surveys that have usually included questions on how people access the internet. The Pew surveys show that broadband adoption at home has grown from 6% in 2001 to 73% in 2019. Its national surveys also permit analysis of the demographic components of broadband adoption, as well as people’s attitudes about broadband’s importance, and the role of broadband access in areas such as lifelong learning. 

Of note is the difference in broadband adoption figures for Pew and the ACS. The 2017 ACS 1-year estimate finds that 83.5% of Americans subscribe to broadband, while Pew’s figure for January 2018 is 65% and, as noted, 73% in 2019. When ACS recorded a 76.7% broadband adoption figure in 2015, Pew’s figure was 67% over the course of the year. One can only speculate as to the reason for the difference. Telephone surveys have been suffering from declines in response rates, which may explain some of the difference. If those who do respond to telephone surveys are less likely to have broadband (e.g., older Americans) and those who are less likely to take telephone surveys are more likely to have broadband (e.g., younger Americans), weighting data to compensate for that may still not fully correct for such under-response. A common concern about telephone surveys is that they rely on landline phones and not (or not enough) on cell phones. The Pew Research Center samples, however, are composed of 75% of respondents who were called on cell phones.

Let the analyst beware

All of this is to say that, when looking at broadband data, the analyst should acquire some familiarity with the methods of data collection and their limitations. None of the data sources discussed above are perfect – but for analysts comfortable with nuance, the perfect need not be the enemy of the good.

John B. Horrigan is Senior Fellow at the Technology Policy Institute, with a focus on technology adoption, digital inclusion, and evaluating the outcomes and impacts of programs designed to promote communications technology adoption and use. Horrigan is also currently a consultant to the Urban Libraries Council. Horrigan served at the Federal Communications Commission as a member of the leadership team for the development of the National Broadband Plan. Additionally, he has served as an Associate Director for Research at the Pew Research Center, where he focused on libraries and their impact on communities, as well as technology adoption patterns and open government data.

Benton, a non-profit, operating foundation, believes that communication policy - rooted in the values of access, equity, and diversity - has the power to deliver new opportunities and strengthen communities to bridge our divides. Our goal is to bring open, affordable, high-capacity broadband to all people in the U.S. to ensure a thriving democracy.

© Benton Foundation 2019. Redistribution of this email publication - both internally and externally - is encouraged if it includes this copyright statement.

For subscribe/unsubscribe info, please email headlinesATbentonDOTorg

Kevin Taglang
Executive Editor, Communications-related Headlines
Benton Foundation
727 Chicago Avenue
Evanston, IL 60202
847-328-3049
headlines AT benton DOT org

Share this edition:

PUBLIC INTEREST VOICES FOR THE DIGITAL AGE