If you are in the business of piloting development policies with NGOs, this chart should be keeping you awake at night. If you like to think about "sustainability" and "scale", about handing over your activities to the government, you need to be really really worried.
Researchers persuaded World Vision and local government in Kenya to both implement that exact same intervention at the same time. The program as implemented by World Vision found a large impact on test scores. The exact same program, as implemented by the local government, found zero impact.
For more see Gabriel Demombynes.
His conclusion:
Evaluation skeptics may try to cite this as evidence that RCTs are a waste of time, since it suggests that successful interventions implemented by NGOs, as they often are in experiments, may not be replicated at scale by governments. Others might take the paper to indicate that NGOs should be the preferred vehicle for interventions. I think these readings would be mistaken, and I take two reflections from the paper. First, we should do many more rigorous studies working with governments where we vary forms of service delivery to better understand what can work in practice. Second, the World Bank’s approach to public services—the long, difficult slog of working to improve government systems—is the right one, because it’s the only way to ultimately make services work for the poor at large scale.I agree. Clearly working through government systems is essential. Innovations for Poverty Action are doing just this - scaling up the exact same contract teacher program tested with NGOs in Kenya and India, but doing it with the government in Ghana, and doing an RCT as they go.
Addendum: Here is the link to the full paper: http://www.cgdev.org/doc/kenya_rct_webdraft.pdf
4 comments:
Thanks for this Lee.
Since it's not easily google-able yet, I just wanted to add a link to the
full paper for the truly interested reader.
http://www.cgdev.org/doc/kenya_rct_webdraft.pdf
Agree
completely that this shouldn't be read as an indictment of RCTs. Everyone involved in this project would like
to see more RCTs in development.
I think
one underlying lesson that probably doesn't come through very well in a graph,
or even our abstract, is about the politics that underlies the
implementation failures here. Most of the exciting stuff (in my mind)
that development economists are doing with RCTs -- things like contract
teachers and other tweaks to accountability and incentives -- are politically
contentious. And while the political opposition to a small pilot is
probably gonna be small, the backlash against the Kenyan government's proposal
to employ 18,000 contract teachers was huge -- court battles, protests in the
street, etc. The program was irreparably damaged in the process, and at
this point has ceased to exist.
Of course we'd like to think there are broader lessons there about how we
design interventions. For right now though, we just want to be careful not to point our finger at the Ministry of
Education and say 'look, they failed'. (Not accusing you of this at all.) There are some deep, structural explanations behind that failure. And there
are some highly talented and motivated people in the Ministry who worked hard
to make the contract teacher program happen and evaluate it rigorously, and who have been unflinching about not hiding bad results as they came in.
Thanks Justin, I agree entirely that the people from the Ministry of Education who were involved in making this trial happen deserve enormous credit.
Thanks much! I have tried to ask a few different questions over here (http://hlanthorn.com/2012/04/15/experimenting-with-intention/) if you are interested. I don't actually offer any insightful answers.
Among them:
- how involved should the putative implementer be in the design & conduct of the experiment?- how much training and capacity building with the future implementer should be built into the experimental process? would we start to consider ethical requirements in this regard (i.e. experimenters have some obligation to train as well, as needed)?- if something doesn't work, what responsibility do we have to help enhance the public sector's (or other implementer's) capacity? i.e. is the response to a null finding a scrapping of the idea or a re-tooling of the implementer? or something else?
Its such as you read my thoughts! You seem to understand so much approximately
this, such as you wrote the ebook in it or something.
I feel that you could do with some p.c. to drive the message home a
bit, however other than that, this is wonderful blog. An excellent read.
I'll certainly be back.
Feel free to visit my blog post: http://wikitesti.com/index.php/Utente:FatimaCam
Post a Comment