Improving the Evaluation of Climate Information Services and their Impacts in Africa: Assessments in Rwanda and Senegal

It is challenging to evaluate the impact of climate information services (CIS) because the information they provide has value only when users can act on it. For example, the most accurate seasonal precipitation forecast may have tremendous value to a man whose responsibility it is to cultivate rain-fed grains and who owns his farming equipment. A man who has to borrow farming equipment may value the forecast but lack the capacity to act on it in a timely manner. And this same forecast may have little to no value for women in either household if they are not allowed to make decisions about rain-fed cultivation. As this example shows, the pathway from delivery of information to observed outcomes, such as increased yields, are influenced by complex and often context-specific social and economic factors. These factors shape who makes what livelihoods decisions, what goals they seek, and what information they are able to access and use.

These two reports employ the Livelihoods as Intimate Government (LIG) approach, an innovative qualitative livelihoods approach, to navigate these factors and identify pathways of uptake, use, and impact that emerge when CIS interventions meet the needs, roles, and responsibilities of specific end users. 

In both countries, Senegal and Rwanda, data collection and analysis identified the stressors and shocks prioritized by community members, and used this information to stratify communities into groups with shared sets of reported vulnerabilities. Analysis of these groups provided an understanding of the different livelihood goals and concerns present in each community, informed our understanding about the logic of potential CIS users in specific livelihoods zones, and yielded ways in which climate information might inform different users’ activities and decisions. The reports examine the livelihoods of potential CIS users in two countries, Senegal and Rwanda, spanning three FEWSNET Livelihood Zones. The analysis and findings of the two country studies are also outlined in a concise summary report.

Beyond describing likely pathways for CIS uptake in each of these three livelihoods zones, each report contributes to our understanding of effective design processes and impact evaluations for CIS. For example, they demonstrate that:

• Focusing either CIS design or monitoring and evaluation at the community level obscures important patterns within that population which can further our understanding of CIS impact. To address this issue, we suggest a two-stage stratification of any user population. First, users should be stratified by shared assemblages of vulnerability. Then, each group produced in the first stage of stratification should itself be stratified by the social characteristics (i.e. gender, age, and ethnicity) that shape individual’s roles and responsibilities vis a vis livelihoods activities. This stratification strategy identifies the decision-makers for climate-affected activities, and the different ways in which different decision-makers might use climate information to inform these activities.
• Identifying impact first requires understanding the character of expected impact. For example, an effective CIS might boost yields or it might help farmers avoid losses. Both have value to farmers, but if they are not evaluated with an eye toward different kinds of impact (or even different impacts among different users), one or the other impact might be overlooked.
• Rigorously identifying impact requires longitudinal data to gauge CIS use and impact across a range of seasonal conditions.

The findings of each report are also contextualized in the larger CISRI learning agendas (Users and Needs and Evaluation agendas) to further the wider effort to improve the practice of CIS design, implementation, and monitoring and evaluation.

• Senegal’s Multidisciplinary Working Group CIS Model: A Qualitative Assessment of CIS Users and Their Needs
• Rwanda’s Climate Services for Agriculture Initiative and the Participatory Integrated Climate Services for Agriculture: A Qualitative Assessment of CIS Users and Their Needs
• Summary Report: Innovative Qualitative Approaches for CIS Monitoring and Evaluation