Digital Inclusion and Outcomes-Based Evaluation

In recent years, government agencies, private foundations, and community-based organizations have increasingly sought to understand how programs that promote digital inclusion lead to social and economic outcomes for individuals, programs, and communities. This push to measure outcomes has been driven, in part, by a larger trend to ensure that dollars are being used efficiently to improve lives rather than simply to deliver services. A new report, published by Benton Foundation, describes the challenges facing community-based organizations and other key stakeholders in using outcomes-based evaluation to measure the success of their digital inclusion programs and offers recommendations toward addressing these shared barriers.

This new research builds off Dr. Colin Rhinesmith’s Digital Inclusion and Meaningful Broadband Adoption Initiatives, released in early 2016. That report identified the core offerings of digital inclusion organizations – from providing low-cost broadband, and the devices to connect to it, while helping new broadband adopters gain the skills they need to navigate the Internet and online services. In this national study of digital inclusion organizations, Dr. Rhinesmith also noted that most of the digital inclusion organizations that participated in this study did not have outcomes-based evaluation frameworks. However, all recognized the importance of having them. This finding led us to conduct this deeper research on the challenges surrounding outcomes-based evaluation.

Twenty-some years ago community technology centers offered training and public access to computers (a few with Internet access). Today we have digital inclusion programs provided by community-based organizations, libraries, and local government. The purpose twenty years ago was not the technology but what one could do with it. The same is true today. The difference is that we are now trying to clearly define the outcomes of access and use of the technology. What we do with the technology and the outcomes will continue to evolve as the technology evolves.

In an era of increasing pressure to show funders, policymakers, and constituents the impact of digital inclusion programs, community-based organizations, in particular, face significant barriers in conducting outcomes-based evaluation. We hope this report will help move the field forward by providing readers with a more complete understanding of the complex challenges facing digital inclusion practitioners.

In workshop discussions with practitioners, funders, policymakers, and other key digital inclusion stakeholders, we identified the core outcomes-based evaluation needs of the digital inclusion field:

1. Shared vocabulary. One of the most pressing issues that emerged from the workshops is the lack of shared vocabulary across the field to define “digital inclusion” and “outcomes-based evaluation.” While most workshop participants articulated what these terms mean to their own work, it was more difficult for them to say how these terms are being implemented across the field.
2. Outcomes-based evaluation tools and implementation support. Workshop participants expressed their concern with not knowing where to find and how to use the tools needed to conduct outcomes-based evaluation of digital inclusion programs. Many participants also indicated that staff within organizations would need training to take advantage of the most appropriate, and culturally-relevant, outcomes-based evaluation tools in ways that reflect the specific needs and aspirations of their communities.
3. Need for common indicators. Participants explained they lacked access to common indicators, the measures programs select as markers for their success. Participants in our workshops often expressed concerns about what and how to measure these changes in ways that were also meaningful to the communities they serve.
4. Time and resources. While most participants understood the value of outcomes-based evaluation, particularly how it could help them show the impact of their digital inclusion initiatives, few could dedicate the time and resources to support this work.

Outcomes-based evaluation can be helpful for funders of digital inclusion programs, policymakers, and for the organizations that offer digital inclusion programs. Outcomes-based evaluation can be useful in the program planning and implementation phases, as well as for measuring program outcomes over time. Therefore, this methodology is often iterative and requires organizations to engage in a continuous process of data gathering and analysis.

The report is being released at Net Inclusion 2017 in St. Paul, Minnesota. Digital inclusion community practitioners, advocates, academics, Internet service providers, and policymakers are gathering to discuss current and potential local, state, and federal policies and policy innovations that could increase digital equity; current and potential sources of financial and programmatic support for digital inclusion programs; and share digital inclusion best practices and new strategies from across the US.

Digital Inclusion Outcomes-Based Evaluation is available online at www.benton.org/outcomes-based-evaluation-report