Introduction

Currently, the Government of El Salvador’s Ministry of Public Health and Social Assistance (MPSAS) is in the midst of a major reform of its social safety net. The World Bank helped finance this reform and has required the government to evaluate the cornerstone of the social safety net reform, Red Solidaria (El Rojo for short). The World Bank has selected you to lead the evaluation. Please read the background information and then answer the following questions. You will conclude by recommending the best feasible design to conduct the impact evaluation.

1. Outline a logic model for the program.
2. Propose two researchable questions that can be answered in the evaluation.
3. Propose outcome measures contained in the available data that you could use to address each of your researchable questions.
4. You have been asked to consider and report on four evaluation designs. They are:
a. Single-group interrupted time-series design
b. Unequal comparison group pre- and post-test design
c. Randomized control trial

For each design:
a. Describe how the evaluation would be conducted. Include a description using the R X O design diagram.
b. Discuss whether the design can be employed given the program structure and available data. If it cannot, describe why not.
c. Evaluate the internal and external validity of the design. What are its strengths and weaknesses?

5. Which design would you select to evaluate El Rojo? Justify your recommendation based on the strength of the design and feasibility.

Background on El Rojo

In early 2006, the Government of El Salvador undertook a reform of its social safety net system, refocusing the system around El Rojo, a conditional cash transfer program. The primary objective of the program is to link social assistance with human capital accumulation. In 2006 nearly 40 percent of Salvadorans were living under the poverty line (earning less than $1.25 per day) and favorable social indicators masked a significant lack of access to education, especially among the poor. For instance, although poor children are typically enrolled in school, they sometimes do not attend regularly.

Through the conditional cash transfer program, eligible families received cash assistance conditional on regular attendance at school and regular checkups at health centers. All beneficiaries continued to receive the assistance as long as they continued to meet the program’s conditions. The conditions for receiving benefits were as follows:

• Children 0-6 years old need to visit a health clinic every two months during the first year and twice a year thereafter.
• Children 7-17 years old need to attend school at least 85 percent of school days as well as attend health clinic twice a year.

The cash transfer amount given per eligible member of a household was $12 per month (later increased to $15) and the average amount received per household was $25 per month.
The El Rojo program gathered detailed information about applicants and ranked them by level of poverty. Each household interested in the program would fill out an application form, reporting information on socio economic, education, health, and demographic characteristics. This information was entered into a computer system that used an algorithm to analyze each household’s responses and calculate a household eligibility score. If the score was below a pre determined threshold, then the household was considered eligible, and invited to register in the program. If the score exceeded the threshold, then the household was not eligible for the program. Applicants were generally informed of their eligibility status about 2 months after applying and received benefits within 4 months of applying.

The program was piloted in the region of Tacuba in 2006, and was expanded to the 39 poorest municipalities countrywide in 2008. The Ministry of Public Health and Social Assistance started to accept applications July 1, 2006 and continues to accept applications in participating municipalities. In 2007, approximately 35,000 households applied for the program and about 20,000 households were eligible and registered for the program. Most Rojo eligible participants are very poor. At the time of the 2006 application, about 4/5 of Rojo participants fell into the bottom three quintiles of the consumption distribution, which means they were among the lowest 60 percent of Salvadoran households in terms of income.

Is El Rojo Effective?

Even though many in the Ministry were certain that El Rojo would help the poor, there were many unanswered questions. Officials were not certain how poor households would react to the cash grants, whether the grants were sufficiently large, or how easy it would be to change long-held patterns of behavior.

The following data are available for use in the evaluation.

• El Rojo Application: This application contains household (and household member) data identified by the head of household’s national identification number (like a US social security number). The data include socio‐economic, housing, education, health, and demographic characteristics for all applicants at the time they applied to the program. This includes school and health records from the past year. Applications are accepted on a rolling basis from 2006-2011.

• 2010 El Rojo Database: Data compiled on households currently enrolled or previously enrolled in the Rojo program since its inception. These data are also identified by the head of household’s national identification number (like a US social security number). The data include school and health clinic records as well as the date of Rojo enrollment and total benefits received. Database compiled in June 2010.

• 2011 El Salvador Survey of Poor Children (SPC): This is an anonymous survey that contains data on a random sample of Salvadoran households in high poverty areas. A “high poverty area” is defined as an area where more than half of its residents are below the poverty line. Public employees from the MPSAS surveyed parents and children in their homes collecting information on housing conditions and socio-economic characteristics. They also collected school records and health visits from the previous 10 months. The survey includes information on whether the child is enrolled in the El Rojo or not. Total Sample Size: 25,000 households. Conducted in June 2011.

Introduction Currently, the Government of El Salvador’s Ministr...