What is impact?

Everyone is talking about effect and impact. For some years now, even in the non-profit, donor-funded social sector in Germany. Some people talk about impact so much that their audience rolls their eyes when the word is mentioned. One reason for this reaction may be overwork. But more than anything else, it is due to the still unclear and inconsistent use of the term.

What is impact or effect? First of all, effect is the second half of causality. Cause is the first half and the two are inextricably linked. In the social sector, effect or impact refer to the medium and long-term changes that occur in a specific target group, in their living environment and ultimately in parts of society as a result of a specific service. A causal cause-and-effect relationship is therefore assumed between a social intervention and these changes.

For a long time, little attention was paid in Germany to whether and how social programs actually worked. The assumption of effectiveness and, in particular, the presumed good will of all those involved were sufficient. However, there is a growing realization that interventions may not only not work as intended, but may also be completely ineffective, miss the target group entirely or even be harmful. It should therefore go without saying that actors in the social sector should do everything in their power to reach as many people in their target group as possible with their work and that this should be proven in the best possible way. All the more so as this work is largely financed by taxpayers’ money. The current attention to impact, as reflected in the popularity of the Kursbuch Wirkung (Course book on impact) by Phineo is therefore a good sign. But only if the underlying terms and concepts are dealt with in a differentiated, consistent and honest manner.

An assumption is just an assumption

What is necessary to be able to talk about the impact of a program? First of all, you should know what you actually want to achieve with a program, i.e. have concrete and detailed project goals. And then check whether and how the program in question achieves precisely these objectives.

What could such a review look like? It seems that many foundations, nonprofit organizations, and consultants are still gnashing their teeth over this question - especially in Germany, while other countries have long since moved on. It is actually quite simple: the causality between cause and effect shows the way. As long as I cannot prove that a change in the target group has actually taken place as a result of my social intervention, I cannot and must not speak of a proven effect. However, scientifically valid proof of a causal relationship between intervention and change is not easy to provide. Scientific studies cost money and tie up human resources. The new attention to impact therefore faces significant barriers. Impact is not trivial. This helps to explain why surprisingly few organizations actually want to prove their impact.

What can be done to ensure the delicate flame of effectiveness does not fade before it can ignite into a true fire? The Landespräventionsrat Niedersachsen (LPR) provides a notably clear and differentiated approach to the topic of effectiveness with its project “Communities That Care – CTC.” In its recommendation list, the “Grüne Liste Prävention (green list prevention)”, published over 10 years ago, only prevention programs that meet a series of clearly defined requirements are included. This is based on the theoretical model by Jan Veerman and Tom van Yperen. In this model, social interventions are placed on a developmental pathway: At most developmental stages, only methods with descriptive, theoretical, or indicative significance can be used to analyze the programs. However, these methods can only serve as precursors to the actual proof of a causal effectiveness relationship. The lower the developmental stage of the program, the more freely the evaluation methods can be chosen. However, for the proof of a demonstrably causal effectiveness, only methods that truly allow for causal statements are suitable.

Accordingly, the programs included in the prevention list are classified into three categories on the basis of a number of criteria - in addition to concept and implementation quality, these include the level of evaluation, strength of evidence and evaluation results:

1. Effectiveness is theoretically well founded
   
2. Effectiveness is probable
   
3. Effectiveness is proven

Such a differentiated approach would be good for the effectiveness debate in Germany as a whole. It should be the benchmark for all those who currently talk about effectiveness. For just as it is clear that impact can only be claimed if it is causally proven, it is also clear that many organizations will hardly ever be able to provide clear evidence of impact, often because they are simply too small. They lack the human and financial resources to commission and oversee an external study.

A little honesty would do everyone good

In such cases, funders should stop demanding evidence of effectiveness where none can exist. Organizations should cease claiming evidence of effectiveness when there simply is none. Instead, all parties should support the organizations in their development so that they can “formulate their assumptions about causal relationships more explicitly so that they can (…) be subjected to tests \[with causal evidential power]” (Grüne Liste Prävention, evaluation criteria). Or, they should acknowledge that they are funding an organization in good faith that is structurally unable to prove its effectiveness.

If the goal is to provide evidence of effectiveness within a scientifically recognized research design, the term “proof of impact” remains valid. Anything that does not withstand scientific scrutiny due to the lack of causal evidence should be named differently. Perhaps the term “impact orientation” can help here – as a description of the alignment of organizations that have already defined goals on which impact measurement could be based. However, such an impact-friendly attitude alone does not justify – any more than an impact-friendly organizational and process structure – the statement “The program works!” After all, one would not claim that someone who merely intends to save energy someday is already contributing to climate protection.

But even if impact measurement is successful and a program can demonstrate that it is achieving the desired positive effects, there is another critical question that needs to be answered: Are the effects being achieved at a reasonable cost? In other words: Is the program not only effective, but also efficient? After all, the question of how best to reach as many people as possible is always a question of whether, given limited resources, we can achieve that impact for as little money as possible.

Only when both questions are answered - the question of impact and the question of cost per impact achieved - will the foundation be laid for public and private funders to make decisions based on solid evidence rather than friendly hallucinations. This would pave the way for a world in which only those who can prove their impact are allowed to talk about it and receive the funding their work deserves.

(This text was published in German with minor changes in 2017 on the blog of the no longer existing Benckiser Foundation Future.The co-author was Ute Volz. We have decided to publish the text again at Effektiv Spenden because, unfortunately, it is just as relevant today as it was back then)

About the author

Stefan Shaw

Founder & Director of Donor Advisory

Book a meeting

I am one of the founders of Effektiv Spenden. If you have any questions or suggestions, please feel free to send me an email. I will get right back to you.