Measurement metrics for mobile broadband services

Description

On June 1, 2010 the Federal Communications Commission asked for public comment on whether and how to pursue a measurement program for mobile broadband services given the growing significance of mobile internet access. Additionally, the seeks comment on how providers can improve voluntary self-reporting of network performance and coverage.

Measurement metrics for mobile broadband services: We seek to understand the best metrics to measure the performance of mobile broadband services. Performance of mobile broadband networks is becoming more important as mobile broadband plays an increasingly important role in our lives and in our economy.

1. What are the best measurement metrics for mobile broadband services?
a. What performance characteristics should be tracked for mobile broadband networks (e.g., typical data throughput, signal strength, accessibility, retainability, latency, other quality of service parameters)? At what level of temporal and geographic granularity?
b. What parts of the network should be measured? What starting and ending points (e.g., radio access network, middle mile) are most useful and actionable for consumers, regulators and providers?
c. Should measurement processes and standards for mobile broadband services be different than those for fixed broadband connections?

User-generated and other data gathering methods: We seek comment on methods to gather better data for mobile broadband network performance and coverage.

2. What are the best methods for collecting data on mobile broadband performance and coverage for end-users?
a. What are the best available tools in the market today for measuring mobile broadband performance and service coverage?
b. Are there current data sets already available that could be useful for facilitating better consumer disclosures on mobile broadband performance and coverage?
c. Are there existing technologies that can measure actual end-user experience on mobile broadband networks? If so where could the measurements take place (e.g., on the device, inside the network)?

3. How can user generated data (i.e., 'crowdsourcing') on mobile broadband network performance and coverage be utilized to assist in collecting data and improving transparency?
a. What efforts and technologies currently exist that can enable device level data collection on performance and coverage of mobile broadband networks? What metrics could a device level software application collect that could measure mobile broadband performance and coverage (e.g., signal strength, data throughput rate)? What other data points would be valuable to collect in association with that data (e.g., location, tower ID, handset type)?
b. For collecting device level data, what impact does the type of device (e.g., smartphone, feature phones, laptop, wireless modem) itself have on end-user experienced network performance? How, if at all, could a measurement methodology take variations resulting from device type into consideration?
c. How could measurement methodology account for variations in performance due to the location (e.g., basement of house vs. above ground) or movement (e.g., user on a train) of the end-user? How can we account for differences in location determination methods (e.g., GPS) across handsets and providers, if any? How should buildings, topography, weather, continued network build-outs, and other service availability variables be accounted for in the methodology?
d. Can a statistically robust sampling method correct for the variables described above, such as the impact on performance and coverage measurements of movement, device and location variability?
e. How can we measure performance with minimal impact on the network itself? For example, how can active measurement techniques that generate additional network traffic mitigate potential increases in congestion?

4. What are the benefits and costs of measurement for providers, regulators, customers and others?
a. What are the benefits (e.g., transparency, better data, network and international comparability, benefits for researchers, verification of National Broadband Map grantee data)?
b. What are the costs (e.g., hardware costs, usage of the network, consumer hassle, accurate information already exists)?
c. Are there any legal, security, privacy or data sensitivity issues with collecting device level data? If so, how can these issues be addressed?

Publication and communication: We seek comment on the best methods for publishing and communicating mobile broadband network performance metrics to consumers to help them make informed choices about mobile broadband services.

3. How could information on mobile broadband performance and coverage be better communicated to consumers?
a. What are the current best practices for displaying or communicating mobile broadband performance and coverage to consumers today?
b. Are consumers currently being provided with enough accurate and detailed information about performance and service coverage to make informed choices between different mobile broadband network providers?

Current mobile broadband network performance and coverage disclosures: Existing voluntary disclosures related to mobile broadband performance and coverage have proven valuable for consumers. Providers of mobile broadband services usually provide coverage maps and 'up-to' or 'typical' data throughput rates. Third-parties also provide and compile coverage maps for providers (American Roamer) and consumers (Root Wireless). While existing data on mobile broadband services are helpful, gaps remain. For example, the currently provided 'up-to' or 'typical' data throughput rates are rough estimations of actual performance and some coverage maps provide a binary 'yes' or 'no' reading without accounting for signal strength at particular locations, whereas other maps provide more layered readings (such as indoor/outdoor or 'good'/'better'/'best'). Additional voluntary performance measurements and standards could provide better information enabling consumers to make informed choices about mobile broadband services.

4. What measurements are typically performed by service providers today to track mobile broadband network performance and service availability?
a. What tools are currently available for consumers to check coverage and performance at a specific geographic location by mobile broadband network (e.g., coverage maps), and how accurate are the data for typical outdoor and indoor consumer use?
b. How are data for coverage and service area maps collected, verified and displayed (how compiled, how accurate, how granular)? How are data on mobile broadband performance (i.e., data throughput rates) measured and displayed?
c. What technologies are used to collect such data (e.g., RF modeled coverage, drive tests, network reporting, handset data collections)?
d. Are there any voluntary industry standards that are being used in disclosing mobile broadband network performance and coverage to consumers? How could these be improved (e.g., signal strength or throughput bands to map different levels of service quality)?

Issues

Comment due date: 07/08/2010