Measuring adaptation success

By Nick Brooks, Director, Garama 3C

As the amount of money being spent on adaptation grows, governments, donors and international aid agencies are increasingly keen to develop ways of measuring whether adaptation interventions are working. This presents them with a number of challenges. For a start, how can we measure the success of measures to help people cope with climate hazards that will continue to evolve beyond the lifetime of the average development intervention? How can we measure changes in such abstract concepts as resilience or vulnerability, again to hazards that will continue to evolve for decades (and centuries)? And more fundamentally, what criteria should we use to judge whether adaptation is successful?

Back in 2013, the World Bank’s Independent Evaluation Group criticised existing approaches to the monitoring and evaluation (M&E) of resilience interventions (read also ‘adaptation’), stating that

“Current results frameworks on resilience [adaptation] are not outcome-oriented and risk emphasising spending over results.”

Since then, not much has changed, with most of the large climate funds focusing on the measurement of outputs (i.e. goods and services delivered, activities undertaken, people receiving support) rather than outcomes (short-term changes resulting from an intervention) or impacts (longer-term changes to which an intervention contributes). While some results frameworks seek to measure resilience in terms of standard development indicators (e.g. of nutrition, climate-related losses, etc.), they fail to grapple with how changes in these indicators should be interpreted in the context of changing climate stresses and shocks. In short, these frameworks lack coherent approaches that link adaptation and resilience-building activities with changes in people’s ability to cope with and adapt to evolving climate hazards in the shorter term, and to changes in human wellbeing in the longer term. One major donor recently told me that they were struggling to make the conceptual link between their climate spending and their regular development spending, and needed to work harder to demonstrate the relevance of the former to the latter.

And there’s the rub. Donors are spending more and more on climate-related interventions on the grounds that addressing climate change is vital to securing desired development results (1), but they are struggling to make the link between climate and development (the latter now represented by the SDGs) when it comes to their programme and fund-level results frameworks.

This shouldn’t be difficult – at least conceptually. The ultimate aim of adaptation and resilience-building is to maintain or improve human wellbeing in the face of climate change that might otherwise undermine it. Adaptation success ultimately will be measured in terms of indicators of human wellbeing and ‘development outcomes’, interpreted in the context of climate information/data. The latter is vital if we are to understand whether or not adaptation-related interventions have (individually or collectively) delivered real benefits.

And this is where it does get a bit tricky. So let’s take a step back.

Most development professionals working on climate change issues appreciate that we ‘do’ adaptation in order to secure the development outcomes we want in the face of climate change that might otherwise mean these outcomes are not achieved. The classic narrative of successful adaptation that you will hear from development practitioners is one in which intended development results are achieved, and people’s wellbeing and prosperity increases, despite that fact that the climate is becoming generally nastier. This is a narrative in which adaptation effectively neutralises the negative impacts of climate change.

Of course, in a deteriorating climate this is precisely what we want to achieve. However, we must also recognise that this is just one of many possible adaptation narratives. We should also acknowledge that this might be a somewhat utopian goal, and accept that sometimes adaptation will be palliative. In other words, things might not get better, but adaptation might prevent them getting worse. Even this might be optimistic in some circumstances, over some timescales. For example, development indicators might tell us that aspects of human wellbeing are deteriorating, but we then have to ask ourselves whether the deterioration would have been even greater in the absence of adaptation interventions. Whether such a scenario represents ‘successful’ adaptation is, of course, a matter for debate. But it would be foolish to ignore the benefits of interventions that at least reduce the negative effects of climate change, even if they do not eliminate them entirely.

Climate information is vital to the development of narratives of adaptation effectiveness. Without it, we don’t know how the climate hazards addressed by our adaptation and resilience interventions are evolving. If we don’t know how climate hazards are evolving, we can’t say anything about adaptation to them. For example, indicators representing aspects of human wellbeing that are affected by climate might improve over time. However, this might simply be the result of a reduction in the frequency and/or intensity of those hazards. Conversely, wellbeing indicators might exhibit a deterioration. If this occurs as climate hazards are reducing we know something is seriously wrong, and we should probably rethink our approaches to adaptation and resilience. However, if climate hazards are becoming significantly worse, we need to ask ourselves what would have happened without the adaptation interventions – would the situation have been worse still?

By thinking about possible trends in climate hazards and indicators of human wellbeing, we can identify a number of simple adaptation narratives (Figure 1). In terms of trends in climate hazards there are three broad possibilities: hazards might decline, remain constant (i.e. varying around a flat mean), or (most likely in a changing climate) intensify. Similarly, when we measure trends in human wellbeing using appropriate indicators, we might find that wellbeing is increasing, unchanged, or declining – again, three broad possibilities.

Adaptation-success-matrix

Figure 1. A matrix of possible adaptation narratives, based on trends in indicators representing aspects of human wellbeing affected by climate, and trends in relevant climate hazards (represented by appropriate climate variables or indices)

This gives us nine possible narratives linking the evolution of climate hazards and human wellbeing (Figure 1). At the extremes we have cases in which wellbeing improves despite intensifying climate hazards (the classic ‘successful adaptation’ narrative), and in which wellbeing declines despite improving climatic conditions (‘maladaptation’). In between these extremes we might see improved wellbeing against a backdrop of unchanging climate hazards (better resilience), declining wellbeing against in same constant climatic context (reduced resilience/increased vulnerability), and a number of other variations.

The most difficult cases to interpret are those in which human wellbeing declines against a background of worsening climate hazards, and in which wellbeing improves but against a background of improved climatic conditions. In the former case, we need to ask whether the decline in wellbeing would have been greater without intervention. In the latter case, we must ask whether intervention was redundant, or whether it amplified gains in wellbeing that occurred at least in part due to an improvement in climatic conditions.

To interpret such problem cases we need some sort of ‘counterfactual’ that tells us what would have happened in the absence of adaptation/resilience interventions. This may be based on qualitative information, for example derived from participatory work with the beneficiaries of interventions. Beneficiaries might be asked whether things would have been even worse without the support received via the intervention. In some (likely limited) cases, counterfactuals might be constructed using statistical relationships between climate-related variables and wellbeing/development indicators. For example, where there is a good historical correlation between rainfall anomalies and variations in crop yields or household incomes, this correlation might be used to ‘predict’ expected anomalies in yields or incomes during periods of climatic stress. These predicted or modelled anomalies can be compared with actual observed/measured anomalies. If a predicted deficit in yield or income is greater than the actual deficit, it might be concluded that adaptation has reduced the negative effects of the climate stress in question, provided this interpretation is supported by other evidence such as beneficiary feedback.

In order to develop the above narratives and counterfactuals we need good climate information/data that tells us how relevant climate hazards are evolving. Climate information is therefore vital to the development of successful results frameworks, particularly for tracking adaptation performance in the long-term. Such tracking is increasingly of interest to national governments, an increasing number of which are developing national systems for tracking how successfully they are responding to climate change. While donor programmes tend to operate over much shorter timescales, the establishment of mechanisms for tracking climate hazards, human wellbeing and – by extension – adaptation performance is vital for learning ‘what works’ in terms of programme contributions to longer-term impacts.

(1) For example, the UK has recently upped its spending on non-domestic climate change programmes from around £3.8 to £5.8 billion over a 5-year period, with the increase in spending coming from within its overseas aid budget.

For further discussion of the use of wellbeing indicators and climate information to evaluate adaptation performance, see the following Briefing Note from IIED, written by the author of this blog post as part of the IIED’s ‘Tracking Adaptation and Measuring Development’ (TAMD) project. More general guidance on Adaptation M&E is provided in other TAMD publications, available here.

The issues discussed here are addressed in more detail in our 2-day Monitoring & Evaluation for Adaptation training course, held regularly in Norwich, UK. Further details of our training courses are available here.

Posted in Uncategorized | Tagged , , , , , , , , , , , , , , , , , , | Comments Off on Measuring adaptation success

July training dates

We have confirmed the dates of our July training courses. Our 3-day Adaptation and Mainstreaming course will run from 4-6 July. This will be followed immediately by our 2-day Monitoring and Evaluation for Adaptation course from 7-8 July. Both courses will be held in Norwich, UK.

If you don’t want to wait until July, there are still places on our Spring courses (as above), which will run from 14-18 March.

Further details of all our courses are available here.

Posted in Training | Tagged , , , , , , , , | Comments Off on July training dates

Next training courses

This is a very quick notice to say that our next Adaptation and Mainstreaming course will be held in Norwich from 14th-16th July 2016, followed immediately by our Monitoring and Evaluation for Adaptation course on 17th-18th.

We intend to run these courses back-to-back again in the first (4th-8th) or second (11th-15th) July.

Please see our training pages for further details.

Posted in Uncategorized | Tagged , , , , , , , | Comments Off on Next training courses

A quick update

We were hacked, so apologies to anyone who expected to read about our work on climate change adaptation but was met with a series of dubious adverts for dodgy training shoes. Hopefully this issue has been resolved.

Meanwhile, we are considering dates for the next set of training courses, namely our 3-day Climate Change Adaptation and Mainstreaming course, and our 2-day Monitoring and Evaluation for Adaptation course. The intention is to run these back-to-back sometime in early 2016. We are also considering dates for our 2-day course on Climate Change and Migration. All these course will be held in Norwich, UK, but they, or versions of them, can be delivered anywhere.

If you are interested in our courses feel free to drop us a line and let us know which periods work for you. This will help us identify dates that are suitable for the largest number of potential participants.

Finally, apologies for the fact that this blog has been a bit quiet of late. This has been due to some absences over the summer, and an otherwise very heavy workload. Right now we’re getting stuck into two long-term DFID programmes – the Climate Smart Agriculture Programme (CSAP) for East and Southern Africa, for which we’re leading on M&E and log-frames amongst other things, and the International Climate Fund Monitoring, Evaluation and Learning (ICF-MEL) programme, for which we’re heading up one of three outputs (reviewing and refining the ICF’s Key Performance Indicators). In addition, we’ve been very busy delivering tailored training courses to ITAD, ODI and the Government of Mozambique. Right now our Director is in Jakarta talking about the Tracking Adaptation and Measuring Development (TAMD) framework at the Asia-Pacific Forum on Climate Change Finance and Sustainable Development.

Watch out for a Garama-penned blog post on measuring resilience on the ODI website soon.

Posted in Uncategorized | Tagged , , , , , , , , , , , , , | Comments Off on A quick update

Adaptation, mainstreaming and M&E courses

We still have some places left on our February training courses – the 3-day Adaptation and Mainstreaming course (9-11 Feb) and the 2-day Monitoring and Evaluation for Adaptation course (12-13 Feb), both of which will be held in Norwich, UK. Further details and information on how to register are available here.

You can download the course programmes here:

We intend to repeat these courses in mid-July 2015. If you require any further information about these or other training courses, feel free to email us.

See our previous two posts for more details about some of the recent work that will be covered by the courses, particularly the Adaptation M&E course, and what the courses intend to achieve.

Posted in Uncategorized | Tagged , , , , , , , , , , | Comments Off on Adaptation, mainstreaming and M&E courses

New Guidance for adaptation M&E

Since 2012 Garama has been working with IIED on the development of the Tracking Adaptation and Measuring Development (TAMD) framework for the monitoring and evaluation of adaptation. The aim of TAMD is to provide a framework that can help development planners and project/programme staff to track and evaluate the effectiveness of adaptation activities.

TAMD has already been piloted in a number of countries, including Cambodia, Kenya, Nepal and Pakistan, and is currently being tested out in a number of other countries. Working papers, briefings, country reports and other documentation are available via the IIED website’s TAMD page.

TAMD_Manual_coverThis week has seen the publication of a major TAMD output – a step-by-step manual for the monitoring and evaluation of adaptation. This guidance has been written by Nick Brooks (Garama) and Susannah Fisher (IIED) with additional input from a number of  other co-authors. Tracking Adaptation and Measuring Development: a step-by-step guide can be downloaded here.

Nick Brooks, Garama’s director, has also written a short Briefing on Indicators for the Monitoring and Evaluation of Adaptation, published by IIED and Garama and available for download here.

December 2014 also saw the publication of a further eight TAMD Briefings and Reports addressing methodological issues and country case studies. These can be downloaded from the IIED website here.

The forthcoming Garama-IIED 2-day course on Monitoring and Evaluation for Adaptation, which will run on 12-13 February 2013 immediately after Garama’s 3-day Adaptation and Mainstreaming course, will draw on these materials. Further details of these courses can be found here.

Posted in Uncategorized | Comments Off on New Guidance for adaptation M&E

Adapation, mainstreaming & M&E training courses

Garama will be running its 3-day Adaptation and Mainstreaming and 2-day Monitoring and Evaluation (M&E) for Adaptation training courses back-to-back during the week of 9-13 February 2015.

The Adaptation and Mainstreaming course will give participants an overview of key issues and current debates in adaptation, as well as a practical knowledge of methods for mainstreaming adaptation into development work at different scales. It will include  updates on the science of climate and current thinking on adaptation, content on existing mainstreaming tools and guidance, climate screening of development interventions, climate risk assessment, the selection of adaptation options, and an overview of monitoring and evaluation (M&E) for adaptation. Participants will undertake a number of practical exercises. By the end of the course participants should have sufficient understanding of mainstreaming to pursue it in their daily work, whether in the context of individual projects and programmes, or more widely in the mainstreaming of climate change adaptation into planning at the institutional or governmental level.

The Monitoring and Evaluation (M&E) course, developed in partnership with IIED, will give participants a knowledge of the state-of-the-art of adaptation M&E – an important and rapidly development field in climate change adaptation. Participants will learn about, and apply, recently developed frameworks and methodologies for adaptation M&E, including the Tracking Adaptation and Measuring Development (TAMD) framework and methodologies for measuring resilience. Participants will learn how to undertake the M&E of adaptation at different scales and in different contexts, including at the project and programme level, the levels of national and local government, and in more general institutional contexts.

More details on these courses, including logistical information, trainer details and information on how to register, can be found here. Participants wishing to register for both courses will receive a discount of 10%.

This will be the third time the Adaptation and Mainstreaming course has run, and the second time the M&E course has run in Norwich. In 2014, versions of these courses were delivered for the Swiss Development and Cooperation Agency (SDC) in Nairobi, Irish Aid in Dar es Salaam, and Adam Smith International in Pretoria and London. The courses will be delivered by Nick Brooks and Neha Rai, who have taken a leading role in the development of guidance and methodologies for the mainstreaming and M&E of adaptation, most recently the development of the TAMD framework and M&E methodologies for the UK’s International Climate Fund and the Department of International Development (DFID). As part of the TAMD work, the trainers have also worked on the development of Cambodia’s national M&E system for climate change response, including the delivery of training on M&E to the Cambodian government in a series of workshops throughout 2013 and 2014.

Posted in Uncategorized | Comments Off on Adapation, mainstreaming & M&E training courses

New flyer on 2014 training courses

Here is a handy 1-page flyer summarising the professional development courses we are offering in 2014, in Adaptation & Mainstreaming, Adaptation M&E, and Climate Change & Migration. Please feel free to download and circulate it.

Garama_courses_flyer

Posted in Uncategorized | Tagged , , , , , , , , , , , , , , , , , | Comments Off on New flyer on 2014 training courses

New course on Adaptation M&E

Garama will be partnering with IIED to offer a course on Monitoring and Evaluation for Climate Change Adaptation. Will will run a 2-day version of this course in July 2014 in the UK, in either Norwich, London or Edinburgh (precise dates to be confirmed). After that we plan to run the course regularly in the UK, and to offer it as a mobile course that can be delivered outside the UK and tailored to the needs and interests of development institutions.

The course will outline the challenges of M&E for adaptation, review existing approaches, and present some methodological approaches for assessing the effectiveness of adaptation, that go beyond the usual focus on outputs. The course will include discussion of the TAMD framework (among others) and recent developments in the measurement of resilience. It will address the emerging issue of ‘normalisation’ – i.e. how to track the success of development interventions in the face of evolving climate risk contexts that will make the achievement of development goals more difficult. The course will draw on experiences of developing M&E frameworks in a number of countries, from national-level frameworks to frameworks for tracking adaptation at the district level and smaller scales.

Feel free to send us an email if you want to register your interest at this stage.

Posted in Uncategorized | Tagged , , , , , , , , , , , | Comments Off on New course on Adaptation M&E

Garama’s activities in 2014

2014 looks set to be a busy year for Garama. We are running three instalments of the Adaptation and Mainstreaming course, in March, July and October, with the October course followed immediately by our new course on Climate Change and Migration. In addition, we are consideration running a new course on Monitoring and Evaluation for Climate Change Adaptation – watch this space. Garama is also in discussions with a number of organisations regarding the delivery of tailored training courses, both inside and outside the UK.

Training aside, we are continuing to lead on the methodological of the Tracking Adaptation and Measuring Development (TAMD) framework with IIED. We are currently working on a paper addressing the use of indicators for adaptation M&E within the TAMD framework. In December 2013 Garama’s Director, Nick Brooks, participated in a workshop in Phnom Penh on the development of a monitoring and evaluation framework for Cambodia’s National Climate Change Response System, which will draw on TAMD.

Right now we are working with Landell Mills on a short assignment for DFID, Assessing the impact of ICF programmes on household and community resilience to climate variability and climate change, the aim of which is to (i) evaluate the applicability of existing frameworks for measuring resilience to projects under the International Climate Fund (ICF) and the ICF programme titled Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED) and (ii) develop a methodology for measuring resilience that can be applied in ICF and BRACED contexts.

Posted in Uncategorized | Tagged , , , , , , , , , , , , , | Comments Off on Garama’s activities in 2014