At the MacArthur Foundation Research Network on Opening Governance’s recent meeting at Arizona State University, one particularly area of focus involved designing pathways toward meaningful metrics for open governance initiatives. Although it is unlikely that a universally applicable framework for measuring the impact of implementations of data-driven or collaborative efforts to improve governance can be created, the Network sought to develop the questions that must be addressed toward assessing impact. The suggestions and prompts below provide an early set of issues and decision areas that can inform the gathering of evidence to enable the assessment of whether and how an initiative worked.
The Network’s metrics discussion was informed by the GovLab’s existing inquiry into the challenge of assessing success in open governance initiatives – see the publication “ Toward Metrics for Re(imagining) Governance.”
A first priority when assessing the impact of open governance initiatives, as with all new processes or interventions, involves identifying the delta – if any – from the status quo. With this in mind, prior to any data collection or analysis on the new innovation, establishing a baseline on the current process is essential. Without knowing the impact of legacy processes, it will be impossible to meaningfully discern the level of change brought about by new strategies. Of course, establishing a baseline is often easier said than done, especially in areas in which tradition and inertia are the leading causes for maintaining the current approach, not evidence-based decision making. Complex governance systems, moreover, are not fixed in time and, as such, rarely have a consistent baseline against which to measure change.
Once the baseline is developed, more decisions need to be made and questions to be answered before implementing a system for measuring impact. The mindmap below describes four central decision areas Network members identified that can inform the development of metrics for an open governance initiative, with guiding questions or examples for each.
- Defining the objective of an initiative. What is the ultimate goal of the project? Metrics for an initiative will differ depending on whether the objective is, for example, to involve more people in the functioning of democracy, to change the way government agencies make decisions, to tap outside expertise that can inform decision-making, or to ensure that the buses run on time.
- Identifying the type of impact desired. What would define success for the initiative? Will success be defined by pragmatic measures like money saved, or by normative measures like increased citizen trust? Once the desired impact is identified, specific indicators of impact can be developed – e.g., media mentions of a given intervention as an indicator of increasing public knowledge of new innovations.
- Developing a system of collection and analysis. Is the strategy for collecting and analyzing data on impacts feasible, scalable or being done successfully elsewhere? - **Focusing on the big or small picture.** Is the desired level of impact on a macro or on a micro scale – e.g., should the metrics determine health impacts for an entire city over a sustained period of time, or the impacts on a single hospital’s emergency room?
Building off of these decision areas, the next step for developing a meaningful strategy for assessing the impacts of open governance is the creation of a conceptual framework that connects inputs (like a city’s new open data initiative) to outputs and impact (such as the creation of new data-driven businesses and accordant economic development). And given the fact that open governance initiatives are pieces of much larger processes, such a conceptual framework must also be sensitive to the challenge of separating causality from correlation. The Network is continuing its work in this area with the understanding that, although an all-encompassing system of metrics for open governance is likely not feasible, a conceptual framework could prove useful to those seeking to measure the true impacts of new interventions on governance and, as a result, on people’s lives.
