Synthesis Report: Improving the monitoring and evaluation of CIS to facilitate learning and improve outcomes

The monitoring and evaluation of climate information services (CIS) presents unique challenges. Climate information has no intrinsic value, and therefore can only effect change when its users have the capacity to act on it. Most assessments of CIS identify and measure impact through broad associations between access to/use of CIS and changes in various behaviors and outcomes. While such impact assessments rigorously identify these associations, they cannot differentiate between the various factors that brought about the observed change.

This report seeks to improve the rigor and explanatory power of CIS monitoring and evaluation by identifying pathways of change by which individuals take up and employ the delivered information to produce an observed change in behavior or outcome, and synthesizing it with conventional large-scale survey data to explain changes associated with the implementation of CIS.

In this report, we first describe and then synthesize the findings of two different efforts to identify and explain the impact of a specific CIS, the Multidisciplinary Working Group (MWG) model, on farmers’ behavior and outcomes, using ethnographic data to rigorously interpret survey findings and identify impacts on the behavior of specific users. The first effort, led by the World Agroforestry Centre (ICRAF), is a large randomized survey comparing the livelihoods of those with access to and using the MWG to those without access and who do not claim to use climate information to inform their livelihoods. The second, led by the Humanitarian Response and Development Lab (HURDL) at Clark University, is an intensive ethnographic exploration of livelihoods decision-making that identified likely pathways through which weather and climate information might inform the livelihoods decisions of different residents. Each of these efforts has been described in separate, detailed reports from HURDL and ICRAF (reports forthcoming). Further, we used this pilot as an opportunity to identify the challenges such synthesis presents and suggest means of managing these challenges to enable similar syntheses for other CIS.

The synthesis of two very different types of studies achieved results that could not have been obtained in a single study. There were several important findings from the synthesis:

1. There is clear evidence of behavioral change: access to and use of the MWG model is impacting user decision-making. Though it is a new program, there is already evidence for the impact of the MWG on a variety of livelihoods decisions, crossing all parts of the agriculturalist population of the livelihoods zone examined for this project.
2. Access to and use of the MWG informs and appears to encourage farmer efforts to invest in critical agricultural inputs such as improved seeds and chemical fertilizer. This pattern was most pronounced for the use of chemical fertilizer, and less pronounced in improved seeds.
3. CIS impacts are produced through sustained efforts. The impact of the MWG on the overall material situations of its users is not immediate. Despite the observed changes in decisions and investment, those with access to and using the MWG did not have higher levels of draught animal or productive assets than those without access and not using the information. However, this is a very new program, and durable impacts on the asset situations of users may take several more seasons to become apparent.

The synthesis also drew a number of useful lessons about the evaluation and synthesis processes, and offers a template for future synthetic work:

• Meaningful and careful stratifications of data sets on users and their needs,
• Appropriate measures of uptake and use,
• Measuring impacts and interpreting change,
• Characterizing confidence in and limits on findings, and
• Scalability of findings: FEWSNET livelihood zones provide a plausible spatial scale