CRS Uses the PPI to Test How Variations in Project Implementation Affect Poverty Outreach >

Catholic Relief Services
•05/29/17
• 0 Comments

An interview with Samuel Beecher, Research Officer, Catholic Relief Services

Catholic Relief Services (CRS), founded in 1943 by the United States Conference of Catholic Bishops, seeks to preserve and uphold the sacredness and dignity of all human life, foster charity and justice, and embody Catholic social and moral teachings to promote human development in the United States and abroad. To achieve these goals, CRS engages in the program areas of Emergency Response and Recovery, Agriculture, Health, Education, Justice and Peace Building, Partnership and Capacity Strengthening, and Microfinance.

In partnership with the MasterCard Foundation (MCF), CRS started the Expanding Financial Inclusion (EFI) in Africa program in 2014. The goal of EFI is to expand financial service access to vulnerable households to improve their resilience. To this end, the project aims to create 19,200 new Savings and Internal Lending Communities (SILC) groups with 502,320 members using the Private Service Provider (PSP) methodology in four countries in Africa—Burkina Faso, Senegal, Uganda, and Zambia—by September of 2017. This methodology involves identifying, training and certifying entrepreneurial people as PSPs, who then go on to help members in their community to form and manage savings groups. You can read more about it here.

To date, EFI has formed more than 17,500 groups with 440,000 group members. EFI’s monitoring and evaluation team uses the Progress out of Poverty Index® (PPI®) to measure the depth of poverty outreach among SILC group members and comparison households in the community. Across the four countries, over 75% of these households live on less than $2.50, and between 30-80% live on less than $1.25 per day. The poverty rates for the SILC group members in the four project countries as measured by the PPI are below.  

Country
% Below National
Poverty Line
% Below
$2.50 / Day 
% Below
$1.25 / Day
Burkina Faso 40.1% 85.5% 54.3%
Senegal 52.3% 75.3% 29.8%
Uganda 26.3% 82.8% 45.1%
Zambia 74.8% 97.3% 85.4%

1. When did your organization start using the PPI, and why? What was the need you were hoping to address?

EFI started using the PPI to collect comparison group data at the start of the project in 2014, sampling approximately 50 villages in each of the four implementation countries. The majority of organizations implementing savings-led microfinance programs have a poverty focus, but most face obstacles in reaching the poorest households.  

2. How does your organization use the PPI? To measure poverty outreach? To improve social performance (targeting or product/service design)? To track changes?

EFI is using the PPI to measure poverty outreach, comparing the mean poverty rate of SILC group members to other households in their communities. EFI also uses PPI to determine whether the PSPs are deepening their poverty outreach over time and to test whether variations in the project implementation strategy influence and improve poverty outreach. 

3. Does your organization collect PPI data directly from households, or do you get PPI data reported to you from partners/investees?

EFI works with a network of ten partners in four countries to carry out the training, implementation, monitoring, and evaluation of the project. Initially, EFI hired consultants in all four countries to conduct PPI surveys on a randomly selected group of households in a randomly selected subset of project villages before SILC group mobilization began. Since SILC groups began forming, CRS has trained partner staff members to administer the PPI to new SILC group members in the previously sampled villages using a mobile device with the support of TaroWorks™.

4. Describe the logistics of collecting and using the PPI at your organization.

The EFI project randomly selected 222 villages distributed evenly across the four project countries. CRS hired consultants in all four countries to conduct baseline PPI surveys in the home villages of a randomly selected subset of PSPs before those PSPs began SILC group mobilization. The goal was to have approximately 1,000 reference households in each project country. When administering the interviews, the consultants initially used paper interview sheets and project staff entered the data into an Excel spreadsheet. At a later date, this data was merged with the Salesforce platform.

EFI has offered technical support to its partner implementing organizations in the administration of PPIs using the TaroWorks™ application, exporting the data from Salesforce to Excel. CRS partner staff working in the same project areas administered the same PPI survey to all the households who had at least one member in a SILC group in the PSPs’ home village. To date, approximately 20,000 SILC group members have been administered the PPI survey in the 222 villages and an additional 10,000 PPIs will be administered before the end of the project. The PPIs are collected on a rolling basis: newly formed SILC groups are interviewed within the first two months of group formation.

EFI staff has worked in partnership with Microfinance Opportunities (MFO) to analyze the PPI data. The PPI is a major component of EFI’s monitoring and evaluation plan as it is integrated with all the other research tools and activities which include: Financial Diaries; a standardized reporting system for savings groups called SAVIX; and a tool which CRS designed called the Agent Management table, which tracks PSP performance.

5. Describe the data analysis you conduct on PPI results.

EFI focused on two questions central to the monitoring and evaluation of the EFI project. They are:

  1. What is the mean poverty level of SILC group members, relative to their communities at the start of the program and over time?
  2. What are the effects of the PSP delivery model variants in terms of poverty outreach?

CRS has used the 25,000 PPI surveys (of which 20,000 were SILC group members) collected to compare the poverty rates of non-SILC, comparison households to the poverty rates of all SILC group member households and identified the sequence in which SILC groups in each sampled village were formed. We then analyzed the data to see whether households in groups that formed later were more likely to be poor than those in groups formed earlier.  These analyses allowed us to answer the first research question above. Due to the small sample size of villages with more than 11[1] groups formed, the figure below and the analysis performed includes villages with 11 groups or fewer.

Figure 1: Share of SILC Members in the Poorest Two Poverty Quartiles by Group Cohort

Looking at cohorts 1 through 11 using the village-by-village comparison of SILC members’ poverty rates to the baseline median, we see that the data suggests an increase in depth of poverty outreach in the later cohorts—the average percent of SILC group members in the poorest two poverty quartiles increases the later that the group is formed in the village. For example, on average, the cohort of first groups, when aggregated together had 60% of group members in the poorest two poverty quartiles. By the 5th cohort, the number was up to 68%, and 80% by the 11th cohort. Across all the countries, partners, and groups, approximately 68% of SILC group members were in the poorest two poverty quartiles.

Preliminary analyses have also shown us that different implementing partners have achieved different levels of success in including the poorest members of communities in SILC groups, and four of the ten partners with which EFI is working have established SILC groups whose members have higher poverty rates than the general level in their home area (statistically significant at a 95% confidence interval). The remaining six partners had SILC members with the same, or slightly higher, poverty rates when compared to the comparison households in the village, but the findings were not statistically significant.

6. Discuss the difficulties you faced (or still face!) when using the PPI. How are you solving them?

Operational challenges

Once the consultants finished administering PPIs to reference households, the project equipped field supervisors with smart phones. The TaroWorks application was downloaded to the devices and field supervisors were trained on how to use the application and troubleshoot simple configuration issues. However, the field supervisors faced multiple issues in using the phones and partner staff lacked capacity to fix these problems. Similar problems with hardware have occurred many times in the project and resulted in delays in PPI administration. Identifying, purchasing and delivering spare parts in a timely manner also proved challenging. After two years of issues with the first brand of mobile devices purchased, CRS chose simply to purchase a different brand rather than continuing to order spare parts for the original phones. 

Another challenge which CRS faced in the project was merging data to Salesforce that had been collected on paper sheets. When merged, all of the data was given the Village Name “[Country] Baseline.” CRS worked with MFO to restructure the database and disaggregate the Reference Village Names so that a village-level analysis could be performed in all four countries. 

Technical challenges

CRS faced issues specific to the PPI’s construction and results, particularly around whether the tool is sensitive to variations in poverty in rural African communities. One of CRS’ main goals in using the PPI as a poverty measurement tool was to examine whether PSPs using normal and pro-poor delivery models for SILC groups achieved different levels of poverty outreach. A problem emerged in late 2016 when it became clear that many respondents to the PPI surveys in Zambia in the ‘Expanding Financial Inclusion Financial Diaries’ study had the same, very high poverty likelihoods, yet the Financial Diaries data and information from a questionnaire[2] administered alongside the PPI suggested that there were real differences in the poverty levels of households with the same scores.

To assess the PPI’s sensitivity, EFI triangulated PPI scores with household rankings and categorizations based on the wealth ranking exercise and detailed income data from the Financial Diaries study. The results showed that the PPI may have overestimated the poverty levels of the SILC group members, especially households that were categorized as ‘Well-Off’ or ‘Managing’ in the wealth ranking exercise. A paper with more details on this evaluation of the PPI results is forthcoming.

EFI has also trained partners to perform simple analysis to find average poverty rates of households with different demographic characteristics.

7.   Has the way your organization has used the PPI evolved, or does your organization have plans for it to evolve in the future? If so, why and how?

Because of the results of the triangulation exercise in Northern Zambia (described above), we would recommend that CRS supplement the PPI with other tools to verify sensitivity in measurement where this is programmatically of high importance.

 


[1] The number 11 intentionally selected as the sample size of villages with more than 11 groups formed was low and the results were therefore deemed insignificant. 

[2] The additional questions in the survey included indicators in food security, education, and financial inclusion.

0 Comments

Comments