If you are in the business of piloting development policies with NGOs, this chart should be keeping you awake at night. If you like to think about "sustainability" and "scale", about handing over your activities to the government, you need to be really really worried.
Researchers persuaded World Vision and local government in Kenya to both implement that exact same intervention at the same time. The program as implemented by World Vision found a large impact on test scores. The exact same program, as implemented by the local government, found zero impact.
For more see Gabriel Demombynes.
His conclusion:
Evaluation skeptics may try to cite this as evidence that RCTs are a waste of time, since it suggests that successful interventions implemented by NGOs, as they often are in experiments, may not be replicated at scale by governments. Others might take the paper to indicate that NGOs should be the preferred vehicle for interventions. I think these readings would be mistaken, and I take two reflections from the paper. First, we should do many more rigorous studies working with governments where we vary forms of service delivery to better understand what can work in practice. Second, the World Bank’s approach to public services—the long, difficult slog of working to improve government systems—is the right one, because it’s the only way to ultimately make services work for the poor at large scale.I agree. Clearly working through government systems is essential. Innovations for Poverty Action are doing just this - scaling up the exact same contract teacher program tested with NGOs in Kenya and India, but doing it with the government in Ghana, and doing an RCT as they go.
Addendum: Here is the link to the full paper: http://www.cgdev.org/doc/kenya_rct_webdraft.pdf
