How can participatory action research be used to develop a theory of change and an evaluation framework to benefit the digital equity field? 

In Developing a Digital Equity Theory of Change with Tech Goes Home, our research team discovered several findings that we believe can be useful for the U.S. Department of Commerce’s National Telecommunications and Information Administration (NTIA) and state agencies as they work to create, award, and evaluate funding opportunities for digital equity organizations. 

• Measuring broader outcomes is challenging. One of the resounding messages is that it is difficult for community-based organizations to measure the broader social and economic outcomes of their digital inclusion programs. Measuring outcomes is challenging because of the complexities of proving causality, e.g. an individual who takes a digital literacy class experiences positive social, economic, educational, and health outcomes as a direct result of having taken the class. 
• Program evaluation software is expensive. Not only is outcomes-based evaluation challenging, the software required to collect data and show outcomes is expensive. This is because the software requires customization from IT professionals to adapt the technology to meet the needs of digital inclusion and digital equity organizations.
• Evaluating programs that involve community partners requires additional work. Many digital equity programs rely on other community-based organizations for service delivery. This model of service delivery raised several questions related to how organizations can measure the outcomes and impacts of their programs not only for the individuals who benefit from services, but also for the community-based organizations with whom they partner.

In response to these and other insights from our research participants, we offer policymakers the following recommendations to help them understand the outcomes and impacts of federally funded broadband and digital equity programs. We offer additional recommendations to digital equity organizations who will need to gather insights from the community beneficiaries of digital equity grant programs.

Recommendations for State and Federal Policymakers

• The success of broadband infrastructure programs relies on digital equity funding. Community members cannot access the digital world without affordable and reliable internet service, internet-equipped hardware, and hands-on support in making use of it. When evaluating the success of broadband deployment, administering entities must also consider whether communities have access to the digital equity programs necessary to make use of broadband access. 
• Set aside funding that organizations can use to conduct program evaluation. Digital equity organizations have been arguing for years that if funders expect them to show the outcomes and impacts of their investments, the organizations should be compensated to do this work. Because program evaluation is time-intensive—particularly when engaging covered populations—state and federal entities should allocate funding that can be used by digital equity organizations to measure the success of their state- or federally-funded programs as part of contracts.
• Provide technical assistance on program evaluation for digital equity organizations. In addition to funding, state and federal entities should provide technical assistance to support digital equity organizations conducting outcomes-based evaluations. Counting outputs, such as the number of digital skills classes offered or the number of computers distributed, is a much easier task for organizations. However, if digital equity organizations are required to show the mid- and longer-term outcomes, then technical assistance must be provided by state and federal agencies to help local organizations, particularly under-funded nonprofits, with this work.
• Allow and encourage organizations to use government funding to compensate community members for their expertise. State and federal agencies should require grantees who receive funding to implement and evaluate digital equity programs to provide evidence that community members were engaged in determining what the success of these programs looks like. State and federal agencies should also ensure that compensating community members for their expertise be allowable as a budgetable expense. If the current, unprecedented federal funding is truly to make an impact, then those most impacted by digital inequalities must be included as partners in the creation, implementation, and evaluation of these programs. 

Recommendations for Digital Equity Organizations

• Allocate time, money, and intentional effort to capture insights and expertise from community members. Recent discussions in the digital equity field, both at practitioner conferences and in other spaces where digital equity researchers are gathering, have argued that organizations must pay community members to gain their expertise and participation in program evaluations.
• Engage evaluation participants in their native languages. Digital equity organizations that serve learners who speak multiple languages should work to ensure that their needs are considered in program evaluation efforts. These populations are likely to face compounded barriers to both digital inclusion and survey participation, and their experiences are therefore crucial to accurately understanding the impact of programs. 
• Work with funders to balance reporting requirements with sensitivity to participants’ privacy and attention to self-defined measures of success.  Two of the significant challenges digital equity organizations face are the need to gather data to show funders the outcomes and impacts of their financial support—while also showing community members why these activities are mutually beneficial. There can be additional challenges in marrying what a funder requests and what the organization knows is a more representative indicator of success. Tech Goes Home, for example, has worked with funders to identify accurate measurements of success and establish respectful guidelines to ensure that program evaluation is compelling, representative, and responsive to community needs.

Based on our analysis of qualitative data collected through multiple perspectives, we believe our findings have significant implications for states, municipalities, territories, and tribal communities, particularly as NTIA and states develop requirements for their grantees to evaluate the outcomes and impacts of broadband and digital equity grant programs over the next five years.

Colin Rhinesmith (he/him) is the Founder and Director of the Digital Equity Research Center at the Metropolitan New York Library Council. He is also a Research Fellow with the Quello Center for Media and Information Policy at Michigan State University and Co-Editor-In-Chief of The Journal of Community Informatics. Previously, Dr. Rhinesmith was an Associate Professor and Director of the Community Informatics Lab in the School of Library and Information Science at Simmons University. He has been a Google Policy Fellow and an Adjunct Research Fellow with New America’s Open Technology Institute, a Senior Fellow with the Benton Institute for Broadband & Society, and a Faculty Associate with the Berkman Klein Center for Internet & Society at Harvard University.​

Sangha Kang-Le (she/her/hers) is the Advocacy Research Specialist at Tech Goes Home, a Massachusetts-based nonprofit working to advance digital equity. Before joining TGH, she organized for political campaigns and conducted research on workplace gender equity in East Asia. She also served on the leadership team of the Asian American Women’s Political Initiative, the country’s only political pipeline for AAPI women, and consulted on the development of the Bridge Forward Fund, an equity-focused economic mobility fund. Sangha holds a B.A. in Political Science and East Asian Studies from Simmons University.

Malana Krongelb (she/her) is a research associate with the Digital Equity Research Center at the Metropolitan New York Library Council. A librarian by trade, Malana serves as an award juror for the Coretta Scott King Book Awards and as a council member on the ALA-APA Standing Committee on the Salaries and Status of Library Workers. She holds a B.A. in Ethnic Studies from Brown University, where she founded the nationally recognized Malana Krongelb Zine Collection, as well as an M.S. in Library and Information Science from Simmons University. 

The Benton Institute for Broadband & Society is a non-profit organization dedicated to ensuring that all people in the U.S. have access to competitive, High-Performance Broadband regardless of where they live or who they are. We believe communication policy - rooted in the values of access, equity, and diversity - has the power to deliver new opportunities and strengthen communities.

© Benton Institute for Broadband & Society 2023. Redistribution of this email publication - both internally and externally - is encouraged if it includes this copyright statement.

For subscribe/unsubscribe info, please email headlinesATbentonDOTorg

Kevin Taglang
Executive Editor, Communications-related Headlines
Benton Institute
for Broadband & Society
1041 Ridge Rd, Unit 214
Wilmette, IL 60091
847-220-4531
headlines AT benton DOT org

Share this edition:

Broadband Delivers Opportunities and Strengthens Communities