A lot has been said about evaluation and the impact of the Millennium Villages, most of which boils down to:
"What is the impact of the Millennium Village package of interventions on the area in question?"
The really depressing part though is that this is actually the least interesting question. Chances are that throwing in a whole bunch of extra inputs to a community will create some outputs, and some impact. The whole point of the Millennium Villages though is to provide a model for the rest of rural Africa to follow. The really interesting question is whether African governments have the desire and capability to manage a massive and complex scaling up of integrated service delivery across rural Africa.
A point which basically belongs to Bill Easterly.
How feasible would it really be to replicate something like this on a national level in Ghana? How exactly would it work? Do the
systems of accountability and capability exist at local levels to manage all of these projects? How would coordination and planning work between national ministries and their sectoral plans, and local level priorities?
"What is the impact of the Millennium Village package of interventions on the area in question?"
The really depressing part though is that this is actually the least interesting question. Chances are that throwing in a whole bunch of extra inputs to a community will create some outputs, and some impact. The whole point of the Millennium Villages though is to provide a model for the rest of rural Africa to follow. The really interesting question is whether African governments have the desire and capability to manage a massive and complex scaling up of integrated service delivery across rural Africa.
A point which basically belongs to Bill Easterly.
Mr. Easterly argues that the Millennium approach would not work on a bigger scale because if expanded, “it immediately runs into the problems we’ve all been talking about: corruption, bad leadership, ethnic politics.”
He said, “Sachs is essentially trying to create an island of success in a sea of failure, and maybe he’s done that, but it doesn’t address the sea of failure.”
Mr. Easterly and others have criticized Mr. Sachs as not paying enough attention to bigger-picture issues like governance and corruption, which have stymied some of the best-intentioned and best-financed aid projects.
A proper randomised evaluation could give you a good estimate of the cost-effectiveness of the island. A difference-in-difference estimate could give you a slightly worse estimate. Doing a fake difference-in-difference with unreliable recall baselines, arbitrarily selected control villages, misrepresented results, and mathematical errors, will give you a pretty awful estimate. But either way, you are missing the main point, which is about scale and replication, and how that works.
The Millennium Village project seems to grasp vaguely at these issues but ultimately brush them under the table. From a MV project report:
An anonymous aidworker writes on his blog Bottom-up thinking
Another challenge in some sites is insufficient capacity of local government to take full ownership of MV activities. This is manifested in unfulfilled pledges to perform mandated roles, unsatisfactory maintenance of infrastructure, and insufficient involvement of local elected officials. MV site teams are addressing these challenges by agreeing to jointly implement interventions targeted at improving the performance of sub‐district governments, increasing sensitization and engagement of local government officials, increasing joint monitoring of MV activities in communities, and developing training plans in technical, managerial, and planning skills for local government officials.Or : "we have no clue how to fix the systemic implementation challenges"
An anonymous aidworker writes on his blog Bottom-up thinking
I’ve noticed around here, normally sloth-like civil servants who won’t even sit in a meeting without a generous per diem rush around like lauded socialist workers striving manly (or womanly) in the name of their country when a bigwig is due to visit, working into the night and through weekends, all without any per diems...
I fear all the achievements of the MVP will wash up against the great brick wall that is a change resistant bureaucracy.None of this is to say that the situation is hopeless. It isn't. In particular there are elements of the Millennium Village package which are proven to be effective, cheap, and don't require complicated systems of governance and accountability. Namely distributing insecticide-treated bednets. Aid money can provide them easily, sustainability is less of a concern than other interventions, and you can buy them right now. Check out Givewell for a rigorous independent assessment (and recommendation) of the Against Malaria Foundation. Probably the single best way you could spend some money today.
4 comments:
Thanks for this post, Lee. It raises at least two questions that I have been thinking & trying to write about.
1. Is it OK for us to do experiments - properly evaluated or not but preferably the former - that have no clear scale-up potential? I suppose the counter-argument is that an experiment could be the proof-of-concept needed in order to attract scale-up partners. Still, I think we need to start to be more selective in our experimental/program design in terms of 'likely buyers' (as well as likely implementers) and being much clearer about what those buyers would want to see in terms of evidence before proceeding.
2. Also, I think we need to carefully consider whether a properly designed experiment would really have us all on-board with the MVPs (which is why I am so happy you wrote this post; the experimental design seems fairly beyond the point), while appreciating that in some ways it represented exactly what an experiment should be -- something about which there is deep and genuine controversy. But experimentation is only meaningful if the experiment will actually move the debate forward; it would have been profoundly better, in my opinion, if Sachs (in this case) had stood up and said to the development community & potential funding and implementing partners: 'I really think I am right on this. What evidence would it take to convince you?'
Besides that, though, the other key question to me is how we are going to start moving towards evaluating 'package' interventions/programs - potentially including those of MVP-complexity - in a way that people believe. It's easier to design a convincing experiment about the distribution of a single product like bednets but it seems that the field needs to start to moving beyond single-product interventions and therefore need to think more seriously about how to make multi-pronged trials convincing enough to move towards scale-up if deemed appropriate.
Experiments were initially supply-driven but I think that is ok and even policy-irrelevant experiments have helped to demonstrate the power of the method, and as demand from policy-makers increases (which I think it is doing) more and more experiments will be policy-relevant.
"How feasible would it really be to replicate something like this on a national level in Ghana? How exactly would it work? Do the systems of accountability and capability exist at local levels to manage all of these projects?"
I think the question is how feasible *was* it to replicate something like this on a national level in *any* country in the past 10 years.
Remember, the vision for the MVs was initially much more ambitious. Sachs said in 2006 that he expected the MVP to have 1,000 villages by 2009. Then reality set in. Now it's 2012, and the MVP has 14 villages. He settled, but not for want of money. The binding factor was (and is) the scale and complexity of trying to implement a "clinical" (as Sachs put it in The End of Poverty), holistic set of interventions (dozens of them) individually tailored to a very, very large number of extremely poor villages. Even implementing this in Ghana would be nearly impossible for one "project" or NGO; if it weren't, it'd be done already.
The Do-It-All type of NGO that the MVP is trying to be is too unwieldy to work for many more villages than one or two dozen. It's much easier to distribute bed nets if that's all you do. Economists figured this out hundreds of years ago -- it's called "division of labor."
(quick correction: it's nearly 100 villages, at 14 *sites*)
Post a Comment