Does ‘skin in the game’ improve the level of play?
The experience of Payment by Results on the Girls’ Education Challenge programme
Girls' Education Challenge
Wednesday, February 15, 2017
The Girls’ Education Challenge (GEC), a £355 million DFID programme, provides one of the largest applications of Payment by Results (PbR) in international development to date. The programme involves NGOs delivering 37 projects across 18 countries, with a PbR mechanism applied to 15 projects.
In a PbR programme payment is linked to performance targets; in the case of the GEC targets were primarily for learning measured by literacy and numeracy tests. If projects achieve their targets they receive what they spend, if they over-achieve they can get a bonus (the PbR “upside”), and if they under-achieve they can lose money (the PbR “downside”).
Figure 1: The PbR model applied on the GEC for 15/37 projects on the programme
We undertook research into the experience of PbR on the GEC through a survey followed up by interviews with project managers, staff on the ground and DFID staff.
How do different parts of an organisation react to PbR?
The rationale behind PbR is that having money at risk – ‘skin in the game’ - should increase accountability and improve performance, either by encouraging the provider to innovate or to prioritise their activities to make them more successful.
Our research found that there was a large amount of variation in terms of how projects treated the financial risk of PbR on the GEC. While central office staff were generally very aware of the risks and incentive from PbR and understood the details of the mechanism, local office staff, usually project managers, knew much less about the specifics. We also found that central office and senior staff experienced a lot of stress from the PbR downside risk, particularly in smaller NGOs.
The vast majority of projects on PbR did not share any financial incentive with their staff. In addition, projects that involved a number of consortium partners often did not share the PbR risk with the smaller partners, protecting them from the financial risk. This meant that the level of PbR risk was magnified at the head office of lead project organisations. We found that they were not able to cope well with this risk.
Did PbR improve outcomes?
Because some providers had a PbR mechanism and others didn’t, we were able to compare whether projects’ behaviour differed as a result.
The biggest test of a PbR mechanism is whether organisations adapt and innovate in response to its presence. We found limited evidence that this was the case. Indeed, some respondents from PbR projects stated that it may have actually constrained innovation by making them more risk averse - they were much more concerned about not losing the PbR downside than in innovating to potentially gain the upside.
In addition, respondents pointed to perverse incentives from PbR, for example to work with less disadvantaged children or to prioritise short-term performance over longer-term sustainable and systemic change.
There were some individuals who cited PbR as a motivation to change when they had interim results showing that their activities were not leading to improvements, but many of the non-PbR projects also did this.
As Barder et al. (2014) put it: “the main purpose of PbR is to enable autonomy for local implementers, which is essential to find solutions to complex problems.” We found that while autonomy was present there were some bureaucratic hurdles. The programme did not operate a “pure” PbR approach but rather a “hybrid” approach in which the majority of projects’ payments were linked to output milestones. Making changes to activity milestones and to budgets could be time-consuming and cumbersome which may have hindered projects’ ability to adapt.
PbR leads to improved evaluation for all projects
While we found a lack of evidence that the NGOs on the GEC were able to cope well with the financial risk and limited evidence of innovation or adaptation as a result of the incentive, there were some other benefits that weigh in favour of PbR.
The presence of PbR on the programme led to the development of the rigorous approach to evaluation that the GEC has applied across all projects, including those not on PbR. The result is clearer evidence of whether interventions are genuinely leading to the core programme goals of the GEC. In this way PbR, via the improved evaluation methods, has increased accountability for results on the programme.
In addition, we found evidence that the NGOs on the GEC largely saw the rigorous evaluation as a “step change” in terms of what they were used to, and a big opportunity to prove that their models and organisations were capable of generating robust results.
Recommendations from our research
Several recommendations emerge from the findings of our research that may be applicable to the design of PbR approaches in other programme contexts, particularly in donor contracting of NGOs as service providers:
- It is critical to consider whether a PbR downside will have more negative than positive consequences. As well as financial risks, particularly for smaller organisations, there is the potential that PbR downside may actively discourage innovation.
- Maintaining a PbR mechanism can provide strong incentives for rigorous results measurement. By having a section of contracts with service providers linked directly to outcomes, programme managers and the donor have a clear priority to ensure rigour and robust measurement.
- A smaller PbR upside could be used in order to maintain the incentive for rigorous results measurement without having the costs and risks associated with PbR downside. Allowing the PbR upside to be channelled to unrestricted reserves can provide a strong motivation for smaller NGOs.
- It is critical to ensure the bureaucracy and time associated with changing project design during implementation is minimised to enable adaptive management to focus on outcomes, particularly in hybrid PbR models (where payments are both linked to outcomes and outputs/activities).
- A PbR incentive payment which is structured around longer term outcomes (measured after the end of intervention funding) could reward projects which are more genuinely sustainable, and encourage more long-term and systemic interventions.