15 Lessons from Impact Assessments



17th Nov 2019

15 Lessons from Impact Assessments

15 Lessons from Impact Assessments

Over the past 10 years we’ve travelled an extraordinary journey

Regarding the first request from a client in 2008: The mining and resources industry asked us to assist them with measuring the impact of their social and community investments. Because of the infrastructure they’ve provided and the enterprises they have supported as well as the capacity they’ve built—we have come a long way. 

We never would’ve foreseen how our own practice, learning, development, and services would change & unfold.

The Numbers

Since 2008 we have:

  • Measured the impact of more than R3 billion of social investment funds

  • Assessed the impact of more than 16 development portfolios from education and health, to sport & housing, from arts & culture to welfare & charity

  • Evaluated the impact and return of more than 900 programs

  • Assessed the impact of more than 800 organisations

  • Developed an indicator library of more than 6000 quantitative and qualitative indicators

  • Identified more than 40 dimensions of impact and return on investment

  • Worked in 9 countries 

  • Assisted more than 25 clients, some of which include the largest multinational social investors in Africa

  • Developed an integrated impact assessment methodology (read more on the Next Generation websitewhich aligns to numerous international guidelines, frameworks and standards

  • Developed a client dashboard that displays impact data 

  • Published and shared our impact management & measured approach. Click here to see our publications on the Next Generation website.

  • Subjected our approach to the review of countless peers, academia and competitors
Our view

Because capacity development and learning are part of our theory of change, this article is presented as a reflection of our work. It’s our hope that it’ll assist practitioners and professionals on their journey to greater impact & return.  

It’s our belief that we can all learn together, transform systems and ensure the sustainability of our sector, but collectively we can have greater impact in the communities where we work.

15 Lessons Learnt

1. Impact must be designed and carefully planned 

Planning impact is hugely important. Impact is grounded in specific objectives and aligned to investors’ intent or vision for impact. 

Most funders have a broad idea of:

    1. how they want to change the world
       
    2. which development area they want to focus on

    3. how they foresee change will happen

But, change never happens in a linear fashion.

What most funders need is to document their assumptions, test them and revisit these assumptions constantly. At the end of a funding cycle most funders are surprised at:

    1. how little change happened

    2. the unexpected ways in which and how the change happened

    3. who was impacted and in what way

    4. the negative and unintended consequences of their actions and interventions

    5. change can happen in numerous ways – most of all – not in the way they envisaged.
2. Achieving impact requires effort, time and the impact journey is messy

The majority of funders don’t consider all the stakeholders involved in the journey. And this happens on a number of levels:

  1. the people/organisations directly impacted

  2. the people/organisations responsible for facilitating the change (through implementation)

  3. any other stakeholders (indirectly) impacted by the ecosystem.  
3. Impact is both positive and negative

Oftentimes investors only consider the positive impact their investments and interventions will bring about. Little regard is given to the unintended and indirect negative implications resulting from their programs.  

There’s a saying in development—for every positive impact there’s a negative impact.  And negative impact is still impact. It’s in the negative consequences that the greatest value lies.  

Not considering, understanding and reflecting on negative impact therefore undermines the value of the intervention. This short-changes the investor as well as the recipients and beneficiaries of an intervention. Additionally, it’s contributing to other organisations making the same mistakes, therefore rendering social development ineffective.

4. Failure and challenges are guaranteed

While funders are doing much to manage risk through strict criteria and funding guidelines and extensive due diligence – they fail to:

  1. capture or act on lessons learnt during implementation or 

  2. use insights from monitoring and evaluation processes 

  3. institute timely pro-active actions based on experience and outcomes. 

All of this leads to repeating costly mistakes and disempowering partners & communities.

5. Embracing failure

Whilst there’s plenty of hype around failure, the facts are; failure is painful, expensive, embarrassing and often non-strategic. Additionally, very few funders and organisations share their lessons learnt, which contributes, at least in part, to an ineffective sector.  

While an impact report provides beautiful statistics of positive impact, the value of learning from failure is never exploited.

6. Transparency and accountability are assumed but never fully realised  

Whilst one could assume that development and social investment is based on both strong personal and organisational values—much more must be done to promote collaboration and a positive, long-lasting legacy.  

Changing political contexts, a connected world and a global operating environment require shared objectives—not singular project delivery. Connecting the dots between outputs and impact as well as those responsible for them, needs to be considered.

7. Simple solutions seldom solve complex problems

Investors often have ambitious goals such as: 

  1. changing behaviour 

  2. providing access to services 

  3. alleviating and reducing poverty 

  4. empowering individuals and communities.

As often as they have these goals, they underestimate what it’ll take to change the underlying systems that contribute to the social challenges in the first place.

In most cases, the causes and challenges they’re trying to solve will require many actors, lots of influence and an enormous amount of resources. Inadvertently, impact assessments seldom indicate that investors achieve these ambitious goals. 

Recognising that they cannot solve a myriad of challenges—and limiting their focus and redesigning programs & interventions—helps in effect to increase impact.

8. Innovation isn’t for everybody

Innovation doesn’t necessarily mean impact at scale. To innovate is difficult because it forces us to leave behind the norms and practices that usually guide us. 

Whilst every funder and organisation may want to claim uniqueness—what works in development—has been created and tested over many decades by many people.  

An inherent adversity to risk; a lack of patience to trial interventions in different locations and contexts, as well as going from pilot to scale, all take a lot of resources. In addition to the resources needed, many stakeholders and a variety of development approaches are required. This delivers a range of results, many of which may not deliver the positive impact desired.  

9. Each to their own  

Organisations are made up of people with diverse aspirations, inspirations and motivations.  Recognising that different viewpoints don’t always produce different points of view means that ultimately, in development all stakeholders must see the same end goal. 

What was originally intended may not necessarily be delivered. In this regard, there’s room to be more flexible when outcomes and impact are measured.  

Very seldom are there merely two or three stakeholder groups impacted by an intervention. In most cases, for change to happen, the people who will be impacted need to be a part of the entire process from design, through implementation, to conclusion.  

Unfortunately in most programs, investors and implementing organisations work on interventions for people and not with people. But when impact is measured, it’s by the people for whom the impact was intended.

10. Impact is only impact if there’s evidence

Impact can only be measured from the people who receive it. Whilst impact strategies and theories of change are useful, they’ll remain abstract if not informed by the investees, recipients and beneficiaries.  

Too much investment is directed at people within communities and too many evaluations fail to test the effectiveness, buy-in and support of these recipients. Too little engagement before, during and after interventions is a reality that must be addressed.  

Without engagement, many investments simply disappear over time. This is owed to a lack of engagement and co-development.

11. Impact is determined by the recipient of the intervention  

How you view the world influences how you view impact. In search of impact, we often find ourselves questioning our own values, beliefs and biases. This realisation has brought a consciousness about how we see solutions to problems as well.  

Fundamentally, the aid and development sectors exist because they solve problems & challenges. Therefore, we’ve come to realise that, for the most part, we exist to focus on solutions before anything else.  

This mindset impedes opportunities to learn from others—especially those who are most familiar with the problems that need solving.  

Consequently, development shouldn’t start with solutions—but rather with listening to the people who are receiving the solution, because this is where impact is evident.

12. Investment and development are two different universes

The paradox of the above statement lies in the imbalance of power between investment and implementation. 

In most cases investors are focused on their intent, bringing about change through their investments; whereas social purpose and implementing organisations are being paid to bring about and ensure the actual impact & change are achieved. 

Measuring impact therefore looks at what changed for recipients of investment. Return on investment is the benefit received by the investor. 

Thus, measuring both is necessary, but too often not addressed in impact measurement and management practice.

13. Impact occurs on many levels

Whilst investment strategy is critically important, impact happens—or doesn’t—because of implementation.  

The careful deliberation and management of implementation is what determines if the envisaged impact will materialise or not. In most cases a lot of time and effort are spent focused on creating the perfect strategy, but too little time & attention is awarded to managing the process of creating the impact.

14. Impact is created by many actors 

Social investors battle within the blurred lines of responsibility. These investors are mostly called upon to fund government shortfalls or to step in where the lack of government services has contributed to the problem that needs to be solved. 

The debate on whose responsibility social development programs are has become more critical in the last few years. Driven in part by a slew of factors, the fact is, investors impact is sometimes reduced by governments. 

Very seldom are the opportunities between government, investors and development agencies adequately addressed or leveraged. The three sectors operate independently from each other—public, private and social sectors. As such, all are impacted differently.  

15. Impact isn’t only the outcome of financial resources or investments

For impact to happen there must be a deep understanding of the context in which impact is to be facilitated and achieved. 

Believing that money is the only thing that can bring about impact is a fallacy. Practitioners and implementing organisations will testify that to bring about change, much more than this is needed. 

The availability of skills, infrastructure and other resources in most cases will influence the outcome of an intervention, and therefore have a direct impact on the impact achieved.