International Development Companies:
Leading the way on evidence and evaluation
"The discipline of development demands a strong practice and use of evaluation as a crucial tool to inform our global development efforts and to enable us to make hard choices based on the best available evidence."
- USAID Administrator Rajiv Shah
Four of the five projects selected by USAID for the agency’s first-ever Excellence in Evaluation honor, awarded in November and the focus of an agency-wide event in February, were designed and implemented by CIDC member companies: Abt Associates, Development Alternatives, Inc. (DAI), Management Systems, Inc. (MSI) and University Research Co., LLC (URC).
With the release of USAID’s Evaluation Policy in January 2011, USAID made an ambitious commitment to quality program evaluation - the systematic collection and analysis of information and evidence about program performance and impact. USAID uses evaluation findings to inform decisions, improve program effectiveness, be accountable to stakeholders, and support organizational learning. Awards were given based on reports that provided the most detailed description of the methodology, so the soundness and rigor could be examined, and whether the findings were clearly linked to the evidence and based on fact, rather than opinion.
Brief descriptions of each assessment conducted by the four CIDC member companies that were awarded the Excellence in Evaluation honor are as follows:
Abt Associates
Extending Social Insurance to Informal Sector Workers in Nicaragua via Microfinance Institutions: Results from a Randomized Evaluation
Abt Associates evaluated an insurance program for low-income workers in Nicaragua. By randomizing incentives to enroll in the insurance program, the researchers were able to get an unbiased picture of the effect of the insurance on health care utilization and out-of-pocket spending. Focus group discussions provided additional insight into what worked and what did not in the program’s implementation.
In Nicaragua, private health care providers used the findings to begin improving health plan offerings to low-income workers. Additionally, INISER, the largest insurance company in Nicaragua, used the findings to develop a comprehensive micro insurance strategy. Results were also published in the journal Health Economics.
The evaluation was conducted as part of the global five-year Private Sector Partnerships for Health-One (PSP-One) project and was funded by USAID with co-funding from the Global Development Network.
DAI
PROFIT Zambia Impact Assessment
Although a new generation of private sector development programs has received significant investment over the past decade, few high-quality impact assessments have been conducted to provide guidance on designing and implementing future projects based on lessons learned. Aimed at economic growth for low-income groups and individuals, these programs often involve promoting the competitiveness of selected industries or value chains in global and domestic markets while increasing the participation of, and benefits to, farmers and other micro and small enterprises.
Contracted by USAID to evaluate the Zambia Production, Finance and Improved Technology (PROFIT) project, DAI measured the results of three Zambian project interventions that involved large numbers of smallholders: retail distribution of agricultural inputs and services, beef and cotton. The goal of PROFIT was to increase multi-sector growth to ensure poverty reduction at the household level, and exemplified this new generation of economic growth projects. Through a longitudinal, quasi-experimental design implemented through a mixed-method approach, DAI offered invaluable insight into lessons learned to improve the design and implementation of the new generation economic growth programs in Zambia and throughout the globe.
MSI
From Aid to Trade: Delivering Results
A team of evaluation experts from MSI and the University of Pittsburgh evaluated and concluded that USAID and U.S. government programs have contributed significantly to trade capacity building (TCB). The report found a clear link between the delivery of U.S. foreign assistance and increases in the value of recipient country exports.
The study separately examined USAID trade assistance focused on private sector export expansion, trade policy reforms, increased participation in trade agreements, and efficiency gains from trade facilitation assistance. The evaluation determined that each additional $1 invested by USAID increases the value of developing country exports by $42 two years later.
MSI’s “From Aid to Trade: Delivering Results” report was also presented this past summer at the WTO’s Third Global Review of Aid for Trade. USAID’s Deputy Administrator for the Bureau of Economic Growth, Agriculture and Trade, Eric Postel, led the U.S. delegation and discussed the evaluation results.
URC
Results of Collaborative Improvement: Effects on Health Outcomes and Compliance
with Evidence-based Standards in 27 Applications in 12 Countries
Through the USAID Health Care Improvement Project, University Research Co., LLC (URC) evaluated the results of 27 health care improvement interventions in twelve countries to determine whether a quality improvement method widely used in the U.S. called collaborative improvement is also effective in low- and middle-income countries.
URC’s research team looked at data on interventions carried out over a 10-year period and measured the magnitude and duration of improvements achieved through collaboratives addressing diverse health services, such as maternal, newborn, and child health, HIV/AIDS care, family planning, and malaria and tuberculosis diagnosis and treatment. A distinguishing feature of the evaluation was that it drew on results achieved in a range of health systems: Benin, Bolivia, Ecuador, Guatemala, Honduras, Nicaragua, Niger, Russia, Rwanda, Tanzania, Uganda, and Vietnam.
The study showed that many health care processes can be readily improved, even in low-resource settings. Of the 135 time-series charts evaluated for more than 1300 teams of health workers, the study found that nearly nine out of ten teams attained performance levels (measured as compliance with standards) of at least 80%, while two-thirds reached at least 90%, even though over half of the teams had a starting performance level of 50% or lower.
“This multi-country study provides the largest body of evidence yet published on the effectiveness of collaborative improvement in lower and middle-income countries,” said Dr. James Heiby, Medical Officer in the USAID Global Health Bureau. “We still have a lot to learn about improving care in these health systems, but the average level of improvement achieved across 27 different settings suggests remarkable potential; much wider use of improvement methodologies like this appears to be feasible.”