The Impact of Evaluation

Note: This was first posted on the IPA blog.

Alanna Sheikh started a bit of a debate last week on the limitations of impact evaluations. She cites Andrew Natsios (a former USAID administrator)

USAID has begun to favor health programs over democracy strengthening or governance programs because health programs can be more easily measured for impact. Rule of law efforts, on the other hand, are vital to development but hard to measure and therefore get less funding.

Lots of things are vital for development, but something being vital doesn’t mean that aid funding is necessarily an effective way of supplying it. Not only that, but something being difficult to measure does not make it impossible. And sure enough, JPAL and IPA have conducted a number of evaluations of governance projects, such as working with the police in Rajasthan, on peace education and ex-combatant reintegration projects in Liberia, and evaluating anti-corruption strategies in Indonesia.

Randomised impact evaluations give the strongest evidence available on a project’s effectiveness. If USAID is beginning to favor projects with evidence of impact that is a good thing. The challenge for governance and rule of law advocates is to prove their impact.

Dennis Whittle of Globalgiving.org adds another limitation:

Formal evaluations, including the gold standard of randomized controlled trials, are not scalable.  We simply do not have the time and resources to do centralized, in-depth evaluations of everything.

This argument is like not bothering with lifeboats if they can’t fit everyone in. Evaluations are crucial if we are going to learn whether or not we are wasting our money. And who knows, we might not be able to evaluate every single project, but if we keep coming up with compelling theories of change and keep replicating our findings in different settings, we could certainly try to evaluate every single intervention.