Rewiring measurement & evaluation, as well as impact measurement in a post COVID-19 world

‘All evaluators must now become developmental evaluators, capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected. This is the current context around the world in general and this is the world in which evaluation will exist for the foreseeable future.’

In these uncertain times, when a pandemic is affecting our professional and personal lives, we are faced with many ‘unknown unknowns’. There are risks and causal relationships that we have yet to imagine, let alone conceptualize. Yet, important decisions must be made now, with imperfect information. We reimagine the future of measurement & evaluation (M&I) as well as impact assessment.

This is an uncomfortable space to be in. It is especially uncomfortable for those of us working in evaluation and impact management functions in the field of development. We typically have the luxury of time, operate within clear theories of change, do our best work when we can mix methods, use multiple sources of data and conduct in-depth interviews with a range of stakeholders. 

The rapid spread of a pandemic like COVID-19 stresses the importance of the availability of real-time and reliable data and calls for us to rewire our systems of how we predict and assess impact. As the implications of the COVID-19 pandemic are felt around the world, few will escape its economic repercussions. In the immediate future, competition for resources will be fierce. 

Not only between countries, institutions and governments, but particularly in the social and impact investment as well as development sectors. 

Already we have seen an enormous amount of resources has been made available and distributed quickly in response to the crises, but very soon a wide range of stakeholders will be asking questions about where the resources went, what difference it made and what impact it had. Restructured, adaptive impact management, measurement and safe data gathering methods need to be developed alongside the release of such funds, in real time. 

Accountability and transparency will be critical to protect reputations. These circumstances push us to rethink how to go about our impact measurement work. It forces us to ask some challenging questions.

Rethinking our practice

The World Bank ¹ states that practitioners will have to consider several aspects specifically, which includes:

From an ethical point of view, evaluation work plans will inevitably need adjustments. Careful consideration of the risk-reward ratio of evaluation activities will become more pressing than usual. First and foremost, where evaluation is a key component for assessing whether and how public health interventions and other priority interventions (e.g. social protection and social safety nets for (poor) citizens) work, evaluators need to be in the fray.

This is not only to be able to collect the best possible data and conduct the best possible assessments to inform decision makers during the crisis, but also to substantiate critical debates that will take place once the crisis is winding down.

Moving forward with evaluation work plans and evaluation design, a conceptual shift needs to take place. In a global pandemic of the proportions that we are currently experiencing, the effects reverberate well beyond the public health sector and the (in-)direct health effects of COVID-19 on citizens.

The global pandemic, the associated government-imposed containment measures, and behavioral changes of businesses and citizens during the crisis may have significant and lasting effects on a whole range of issues that are of societal importance and the well-being of citizens. These include the size and structure of the economy, employment rates, food security, poverty levels, and so on. 

‘Now and in the near future, evaluators will have to reflect on, and factor in, both the direct and indirect causal effects of the pandemic in any type of sector or any type of thematic issue that is subject to planned (and ongoing) evaluations’. 

In addition, as many public resources are being diverted to the crisis, ongoing interventions may not be implemented as designed, as part of the resources may be diverted to addressing crisis-related needs. This has implications on how evaluators should look at such interventions.

Redesigning our practice

The Rockefeller Foundation has noted that in a new post-COVID-19 world ², traditional in-person data collection will be hindered. Real-time analysis of data and on-demand reporting will become a requirement for all types of research and evaluation efforts.

As such, leveraging and combining administrative, transactional and big datasets, like satellite images, household survey data, program administrative data, social media analytics, phone call-center data, the information generated through mobile phones, and internet searches (among others) are going to become key sources of data for research, evaluation and impact measurement specialists.

Data science also makes it possible to collect information over a much wider geographical area, to integrate many different kinds of health, economic, socio-cultural and demographic data (to name just a few) and to track changes over time – making it possible to understand and model the ‘big picture’, in a way that was not previously possible.

Additionally, the computing speed of cloud-based big data architectures and data science techniques like machine learning algorithms will need to become a part of the social science toolkit to meet the rapid need for up-to-date findings. A major challenge for evaluators, impact practitioners and other social science practitioners is to ensure the efficient use of currently available data collection and analysis tools and techniques and to learn lessons from the current emergency to ensure that the full potential of these powerful tools will become available to address future crises.

Rewiring our practice

The process of transformation for adopting and adapting these new technologies will be disruptive. To date, many, but certainly not all, researchers and evaluators have been slower than other practitioners to adopt the tools and techniques of data science.

There are many methodological, economic, organizational, and even political reasons for the slower uptake. For example, the methodologies of training machine learning algorithms to build probabilistic predictive and prescriptive models are not well understood by evaluators and other social science researchers, often making it difficult for them to analyze big data.

There are also important ethical issues that researchers and evaluators raise for using data science techniques, particularly black box techniques that hide experimental and/or social biases. 

Another area of concern relates to the ability of agencies or policy-makers to use big data techniques to collect detailed information on communities and to use this information to make important decisions affecting the lives of these communities - without their knowledge or the possibility of dialog. Challenges to the uptake of big data technologies and analytic techniques are real, but if we build more educational and experiential bridges between researchers, evaluators, and data scientists, they are not insurmountable.

Steps toward intentionally integrating the data sciences and social sciences for more rapid, cost-effective and time-sensitive evaluation and impact findings is paramount to the promotion of social good. With the rapid expansion of big data and analytics, it is time for the fields of program evaluation, impact assessment and data science to come together in order to more rapidly and cost-effectively learn what works, improve social solutions, and scale positive impact as never before.

In conclusion:

The opportunity for rewiring our evaluation and impact measurement practices has never been so clearly defined. In this regard then, the last word goes to Michael Quin Patton again:

The global health emergency is a short-term crisis within the larger and longer-term global climate emergency. This health crisis has revealed both the importance and possibility of systems transformation.

This crisis illuminates the scale, scope, and urgency of systems transformations needed worldwide to create a more sustainable and equitable future. This pandemic is reflecting the fragmented and fragile nature of current systems, inadequate for a just and equitable world. As your work adapts to the current reality, think about how you can bring this larger perspective to bear in your work, to be attentive to gather evidence for, and support the kinds of transformations that may be needed after the pandemic subsides.

Balancing long-term threats to the future of humanity with the urgent demands of short-term, crisis-generated interventions demands in-depth transformative evaluative thinking. Evaluators need to be prepared to contribute to finding and following pathways and trajectories toward transformations for a more sustainable future.  

¹ Source:  World Bank Independent Evaluation Group:  Adapting evaluation designs in times of COVID-19 crises

² Source:  Measuring results and impact in the age of big data:  The Rockefeller Foundation: March 2020