Having reviewed many frameworks in systems change we have shortlisted the 5Rs framework to be the starting point for the evaluation methodology, however noting that it comes from USAID (first published in 2014) which may be rooted in a certain knowledge paradigm.  The 5Rs framework was shortlisted primarily because of its user-friendly nature, that it is easy to remember what the different elements represent; and its relevance to the context of social change. However in discussion with USAID’s 5Rs framework co-author, we recognise that 5Rs framework has various limitations, especially for complex systems, and should be complemented with other tools.

USAID defines the 5Rs as resources, roles, relationships, rules and results:

• Resources: Local systems transform resources—such as budgetary allocations or raw materials or inputs—into outputs.
• Roles: Most local systems involve a number of actors who take on various defined roles: producer, consumer, funder and advocate.
• Relationships: In a similar fashion, the interactions between the actors in a local system establish various types of relationships. Some may be commercial; others more administrative and hierarchical.
• Rules: An important feature of local systems is the set of rules that govern them. These rules define or assign roles, determine the nature of relationships between actors and establish the terms of access to the resources on which the system depends.
• Results: The concept of “results” is expanded to include measures of the overall strength of the local system as well as traditional outputs and outcomes. Applying this framework helps identify strengths and weaknesses in existing local systems and provides a guide to systems-strengthening interventions.

Complementing the 5Rs framework, we are also looking at the six conditions of systems change framework:

We are also looking at group model building, which does not prescribe a particular systems change framework but rather, it is a participatory approach that is widely used to build the capacity of practitioners to think in a systems way.

In the context of evaluation, there are three main stages to consider, We are currently testing the approach below with a few systems change interventions, so we recognise that the approach below is simplistic and requires more details:

Planning

• Use the 5Rs framework, or other systems change frameworks, to define the elements of the system;
• Identify leverage points that the changemaker / funder are addressing;
• Incorporate / transform the leverage points into the Theory of Change model (or an adapted version of Theory of Change model which has been criticised as too linear) – USAID will be issuing guidance on this;
• Consider establishing baseline measures for the current system;
• Brainstorm impact metrics based on leverage points;
• Brainstorm overarching learning questions for the whole system;
• Consider how other factors in the system may impact our activities and intended change, and if possible, consider ways to measure them;
• Consider research methods appropriate to capture the impact metrics and learning questions.

Evaluation

• This stage depends on the methods being used and we are still thinking whether there are particular methods most suited within a systems change context. Such as, Social Network analysis, participatory research methods.

Analysis

• This stage also depends on the methods being used, but a particular challenge we are considering is whether it would be difficult to prove attribution.
• We are exploring contribution analysis as a method and its applicability to systems change evaluation.

A question we are still thinking is the relationship between systems change evaluation and developmental evaluation, agile evaluation, as well as other evaluation approaches. This is the very beginning of a draft, and we are looking to co-create with concrete examples to put this methodology to practice, and adapt it based on practice. The final methodology will be much more detailed than this.

Question for co-creators:

1) What do you think about the tentative methodology?
2) Any we missing any crucial steps in evaluation?

Please feel free to share your thoughts with us by commenting on this blog post, or our social media posts, or e-mail [email protected]. Feel free to engage in the discussions by hashtag #SystemsChangeEval.

In terms of next steps, the systems change evaluation methodology has been a learning journey for us, and it’s taking us longer than we originally envisioned. We are hoping to collate your feedback and also publish a few case studies where we’ve been able to apply the systems change evaluation methodology, to build the practice around this area.