By Nick Brooks, Director, Garama 3C
As the amount of money being spent on adaptation grows, governments, donors and international aid agencies are increasingly keen to develop ways of measuring whether adaptation interventions are working. This presents them with a number of challenges. For a start, how can we measure the success of measures to help people cope with climate hazards that will continue to evolve beyond the lifetime of the average development intervention? How can we measure changes in such abstract concepts as resilience or vulnerability, again to hazards that will continue to evolve for decades (and centuries)? And more fundamentally, what criteria should we use to judge whether adaptation is successful?
Back in 2013, the World Bank’s Independent Evaluation Group criticised existing approaches to the monitoring and evaluation (M&E) of resilience interventions (read also ‘adaptation’), stating that
“Current results frameworks on resilience [adaptation] are not outcome-oriented and risk emphasising spending over results.”
Since then, not much has changed, with most of the large climate funds focusing on the measurement of outputs (i.e. goods and services delivered, activities undertaken, people receiving support) rather than outcomes (short-term changes resulting from an intervention) or impacts (longer-term changes to which an intervention contributes). While some results frameworks seek to measure resilience in terms of standard development indicators (e.g. of nutrition, climate-related losses, etc.), they fail to grapple with how changes in these indicators should be interpreted in the context of changing climate stresses and shocks. In short, these frameworks lack coherent approaches that link adaptation and resilience-building activities with changes in people’s ability to cope with and adapt to evolving climate hazards in the shorter term, and to changes in human wellbeing in the longer term. One major donor recently told me that they were struggling to make the conceptual link between their climate spending and their regular development spending, and needed to work harder to demonstrate the relevance of the former to the latter.
And there’s the rub. Donors are spending more and more on climate-related interventions on the grounds that addressing climate change is vital to securing desired development results (1), but they are struggling to make the link between climate and development (the latter now represented by the SDGs) when it comes to their programme and fund-level results frameworks.
This shouldn’t be difficult – at least conceptually. The ultimate aim of adaptation and resilience-building is to maintain or improve human wellbeing in the face of climate change that might otherwise undermine it. Adaptation success ultimately will be measured in terms of indicators of human wellbeing and ‘development outcomes’, interpreted in the context of climate information/data. The latter is vital if we are to understand whether or not adaptation-related interventions have (individually or collectively) delivered real benefits.
And this is where it does get a bit tricky. So let’s take a step back.
Most development professionals working on climate change issues appreciate that we ‘do’ adaptation in order to secure the development outcomes we want in the face of climate change that might otherwise mean these outcomes are not achieved. The classic narrative of successful adaptation that you will hear from development practitioners is one in which intended development results are achieved, and people’s wellbeing and prosperity increases, despite that fact that the climate is becoming generally nastier. This is a narrative in which adaptation effectively neutralises the negative impacts of climate change.
Of course, in a deteriorating climate this is precisely what we want to achieve. However, we must also recognise that this is just one of many possible adaptation narratives. We should also acknowledge that this might be a somewhat utopian goal, and accept that sometimes adaptation will be palliative. In other words, things might not get better, but adaptation might prevent them getting worse. Even this might be optimistic in some circumstances, over some timescales. For example, development indicators might tell us that aspects of human wellbeing are deteriorating, but we then have to ask ourselves whether the deterioration would have been even greater in the absence of adaptation interventions. Whether such a scenario represents ‘successful’ adaptation is, of course, a matter for debate. But it would be foolish to ignore the benefits of interventions that at least reduce the negative effects of climate change, even if they do not eliminate them entirely.
Climate information is vital to the development of narratives of adaptation effectiveness. Without it, we don’t know how the climate hazards addressed by our adaptation and resilience interventions are evolving. If we don’t know how climate hazards are evolving, we can’t say anything about adaptation to them. For example, indicators representing aspects of human wellbeing that are affected by climate might improve over time. However, this might simply be the result of a reduction in the frequency and/or intensity of those hazards. Conversely, wellbeing indicators might exhibit a deterioration. If this occurs as climate hazards are reducing we know something is seriously wrong, and we should probably rethink our approaches to adaptation and resilience. However, if climate hazards are becoming significantly worse, we need to ask ourselves what would have happened without the adaptation interventions – would the situation have been worse still?
By thinking about possible trends in climate hazards and indicators of human wellbeing, we can identify a number of simple adaptation narratives (Figure 1). In terms of trends in climate hazards there are three broad possibilities: hazards might decline, remain constant (i.e. varying around a flat mean), or (most likely in a changing climate) intensify. Similarly, when we measure trends in human wellbeing using appropriate indicators, we might find that wellbeing is increasing, unchanged, or declining – again, three broad possibilities.
Figure 1. A matrix of possible adaptation narratives, based on trends in indicators representing aspects of human wellbeing affected by climate, and trends in relevant climate hazards (represented by appropriate climate variables or indices)
This gives us nine possible narratives linking the evolution of climate hazards and human wellbeing (Figure 1). At the extremes we have cases in which wellbeing improves despite intensifying climate hazards (the classic ‘successful adaptation’ narrative), and in which wellbeing declines despite improving climatic conditions (‘maladaptation’). In between these extremes we might see improved wellbeing against a backdrop of unchanging climate hazards (better resilience), declining wellbeing against in same constant climatic context (reduced resilience/increased vulnerability), and a number of other variations.
The most difficult cases to interpret are those in which human wellbeing declines against a background of worsening climate hazards, and in which wellbeing improves but against a background of improved climatic conditions. In the former case, we need to ask whether the decline in wellbeing would have been greater without intervention. In the latter case, we must ask whether intervention was redundant, or whether it amplified gains in wellbeing that occurred at least in part due to an improvement in climatic conditions.
To interpret such problem cases we need some sort of ‘counterfactual’ that tells us what would have happened in the absence of adaptation/resilience interventions. This may be based on qualitative information, for example derived from participatory work with the beneficiaries of interventions. Beneficiaries might be asked whether things would have been even worse without the support received via the intervention. In some (likely limited) cases, counterfactuals might be constructed using statistical relationships between climate-related variables and wellbeing/development indicators. For example, where there is a good historical correlation between rainfall anomalies and variations in crop yields or household incomes, this correlation might be used to ‘predict’ expected anomalies in yields or incomes during periods of climatic stress. These predicted or modelled anomalies can be compared with actual observed/measured anomalies. If a predicted deficit in yield or income is greater than the actual deficit, it might be concluded that adaptation has reduced the negative effects of the climate stress in question, provided this interpretation is supported by other evidence such as beneficiary feedback.
In order to develop the above narratives and counterfactuals we need good climate information/data that tells us how relevant climate hazards are evolving. Climate information is therefore vital to the development of successful results frameworks, particularly for tracking adaptation performance in the long-term. Such tracking is increasingly of interest to national governments, an increasing number of which are developing national systems for tracking how successfully they are responding to climate change. While donor programmes tend to operate over much shorter timescales, the establishment of mechanisms for tracking climate hazards, human wellbeing and – by extension – adaptation performance is vital for learning ‘what works’ in terms of programme contributions to longer-term impacts.
(1) For example, the UK has recently upped its spending on non-domestic climate change programmes from around £3.8 to £5.8 billion over a 5-year period, with the increase in spending coming from within its overseas aid budget.
For further discussion of the use of wellbeing indicators and climate information to evaluate adaptation performance, see the following Briefing Note from IIED, written by the author of this blog post as part of the IIED’s ‘Tracking Adaptation and Measuring Development’ (TAMD) project. More general guidance on Adaptation M&E is provided in other TAMD publications, available here.
The issues discussed here are addressed in more detail in our 2-day Monitoring & Evaluation for Adaptation training course, held regularly in Norwich, UK. Further details of our training courses are available here.