Bottom-Up Evaluation and Data Ethics

At the occasion of a discussion on data ethics and bottom-up approaches at Opportunity Collaboration, Bonnie Chiu explored with participants the challenges in producing bottom-up evaluations, and some of the potential solutions.

If you are interested in inclusive approaches in impact assessment, have a look at our new report Inclusion of User Voices.

Amid the growing popularity of impact measurement in the social sector, we should not forget our responsibilities towards those who provide the data we use. If data holds a crucial role in impact assessment, users should be seen as equally central.

Much of impact measurement, however, does not reflect the standards of data ethics. By adopting a top-down data collection approach and failing to make impact reports accessible, actors in the field disempower users – the ones who are supposed to benefit from the investment.

In response to these challenges, impact measurement practitioners need to put the power back into the hands of users. Shifting the balance of power comes with many challenges that can be summarised in three main points:

• The power imbalance between funders, grantees and users – the politics of evaluation, as suggested by a participant – determines who sets up the agenda and allocates adequate (or, too often, inadequate) funding for evaluation, with donors having a dominant position.
• The technical difficulties around evaluation – the science of evaluation – for instance, the challenge to translate social change into metrics and the lack of capacity-building effort.
• The culture of evaluation might need to be developed and highlighted cases in which grantees do not value evaluation as such or do not report on unintended consequences.

The potential solutions set out during the session addressed those three broad areas. Firstly, it is essential that donors and grantees work together in a culture of trust and openness, where the failure of social change programme is accepted, and therefore honest evaluation is valued. On the same note, NGOs have to see evaluation as part of the learning and constant improvement of their programmes; they should be passionate about solving a problem, not implementing a particular solution.

Finally, technology might offer some help, from lowering costs of evaluation through new technologies to enabling end users’ participation through smartphones.