By Bonnie Chiu
At The Social Investment Consultancy (TSIC), we see impact evaluation as a key tool for social sector organisations not only to engage funders, but also to inform organisational improvement and sector learning. Impact evaluation is crucial to understand what works, but equally, what not. As TSIC celebrates its 10th anniversary this year, we have been reflecting on our past decade of practice and will be dedicating a series of blogs to reflect on What Works.
A NPC study from 2012 shows that 52% of charities that have increased their measurement efforts say they did so to meet funders’ requirements. While funders are an important driver for sure, from our interactions with clients, what makes them excited about impact evaluation is definitely the opportunity for organisational learning. Why do we talk about being excited? Feelings/emotions may seem to be too ‘fluffy’ to discuss amid rational conversations about data and measurement. But it is important to discuss them in any impact evaluation conversations with clients for three reasons:
First, impact evaluation is a difficult journey that often requires organisational transformation, and lots of hard work. Feeling excited or passionate about the prospects of impact evaluation gives us the ability to overcome those challenges – so that organisations will do whatever it takes to get to a certain point (i.e. measuring impact well).
Second, it can position impact evaluation not just as a compliance exercise, but a learning and strategic one – integral to everything that an organisation does, rather than something that’s done only once a year, or on the side. Almost everyone working in a charity feels passionate about their cause area, and impact evaluation allows us to be better at solving the problems we care so deeply about.
The third point relates to a broader conversation about statistics. Tim Harford, the award-winning economist and Financial Times economist, recently wrote an article on “how to decipher the barrage of statistical propaganda”. He speaks to this point about feelings:
“my statistical postcard begins with advice about emotion rather than logic. When you encounter a new statistical claim, observe your feelings. Yes, it sounds like a line from Star Wars, but we rarely believe anything because we’re compelled to do so by pure deduction or irrefutable evidence… If we don’t notice and pay attention to those feelings, we’re off to a shaky start.”
He then quotes several studies in political science about confirmation biases, and concludes that “it’s important to face up to our feelings before we even begin to process a statistical claim. If we don’t at least acknowledge that we may be bringing some emotional baggage along with us, we have little chance of discerning what’s true.” Indeed, as human beings, we are not purely driven by numbers and rationality but the stories that underpin the numbers, and feelings.
Acknowledging the emotional baggage in impact evaluation in the social sector is difficult, but is an important conversation. As evaluators, we think that it is also our responsibility to encourage these conversations to take place in a safe space. What is the motivation of the organisation of doing impact evaluation? What are the organisation’s feelings towards it – and do the feelings also differ within the organisation? Confronting the negative feelings, especially fear, and channelling them into productive energy, will help any evaluation project be grounded on a solid start.