Mathematising External Validity

Paul Krugman, Noah Smith, and Bryan Caplan had an interesting debate last week on the use (and misuse) of maths in economics.

A helpful illustration is provided this month by a new paper by Lant Pritchett and Justin Sandefur on external validity and RCTs (handy Charles Kenny summary in BusinessWeek here).

The concept of external validity is pretty simple to grasp intuitively. An experiment might give you a good estimate of the impact of a programme in a certain context, but it can't tell you if the same programme will have the same impact in a totally different context.

This is something which is especially obvious when you are actually working on national policy. When you are a writing a brief for a politician or an NGO on an issue, it would just feel stupid to lead with evidence from a totally different country if there is any data at all from the country you are actually in. Not that studies from other countries are uninteresting, but it is just blindingly clear that there are a lot of differences in political, social, and economic context between countries that might make results from a similar programme quite different, and so to use some caution in drawing conclusions, even from perfectly executed experimental studies.

At yet at the same time, "external validity" can seem like a bit of a hand-wavy rebuttal compared to all of the extensive technical theory around internal validity - whether your study is likely to give a biased estimate of programme impact on the population you are studying. There is a lot of detailed methodological analysis looking at exactly what the causes of bias are in different studies and how this bias can be best avoided. So what Pritchett's and Sandefur's paper does is add some detail to the our understanding of external validity, add some maths, and somehow make the critique seem somehow weightier. I think I still find their empirical examples more compelling than their theory, because I'm slow with maths and more interested in empirics, but nonetheless it does seem important to have that kind of systematic logical thinking through of the detail of a problem.

The bottom line:
- Economath - not totally useless, but you can probably get the intuition without it.
- External validity - an important concern, and sometimes contextual understanding matters more than clean identification - but also a reason for more experiments where possible not less

India wonkwar: Sen vs Bhagwati

This is a guest post by Vinayak Uppal (previously an economist in South Sudan, more recently at DFID India, soon to be OPM), summing up the recent debates between heavyweight economists Amartya Sen and Jagdish Bhagwati. He would opt for reading Bhagwati if you had to pick only one. 

“Is there some action a government of India could take that would lead the Indian economy to grow like Indonesia's...If so, what, exactly? ...The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.”

Thinking about India is precisely what some of India’s best-known economists have been doing. An Uncertain Glory, written by Amartya Sen and long time collaborator, Jean Dreze, and India’s Tryst With Destiny by Jagdish Bhagwati and Arvind Panagriya[1] are two recent books that put the spotlight on Indian development.

Economists, even global leaders such as these, rarely make mainstream headlines. The last few weeks however have seen regular updates to an increasingly acrimonious, personal and bitter spat, being played out in op-eds, letters, interviews and talks all over the world, increasingly with strong political overtones.

So what are they saying, and what is the disagreement over?

Sen argues that India under-performs significantly in a variety of social indicators. It therefore needs to focus on building human capital like health and education first, with such services (largely) provided by the State rather than viewing growth as an end in itself.

Bhagwati's book reads like a capitalist manifesto, making a strong case for the gains that reforms have brought the average Indian through economic growth, and argues that far more reforms are needed to sustain progress.

Like all good debates, there is plenty of disagreement. Sen has stated previously that in an under-nourished country such as India, it is stupid to focus obsessively on growth. He points out how India is now like “islands of California in a sea of Sub Saharan Africa”. Bhagwati takes a far more positive view of growth, highlighting the hundreds of millions of people pulled out of poverty in the decades since growth accelerated, and the rapid improvements in almost all social indicators. 

Sen clearly believes that basic services can, and should, be delivered through State systems, rightly highlighting many of the gains made in states like Himachal Pradesh and Kerala. Bhagwati comes out firmly on the side of cash transfers in many of the social services like health and education. His view of the public sector can be simplistic, seeing almost any money spent through government as a waste.

There remain a number of untouched issues. Both are relatively quiet on the environment. Sen skirts around economic reforms and growth, giving only passing references to crucial issues like infrastructure, labour reforms and increasing job creation. Bhagwati’s views on improving governance are limited, as are his views on how to combat corruption and social exclusion. His book also doesn’t answer the basic conundrum of why India has been unable to provide its citizens with a basic standard of living, despite being the world’s biggest democracy and many of its poorer neighbours performing far better.

It is important in the middle of this heated debate, to remember that they agree on the fundamentals. (Bhagwati clearly disagrees with this, calling Kaushik Basu and Montek Ahluwalia simplistic bureaucrats for suggesting such a thing!) Neither of them is debating the importance of growth or questioning the need to spend resources on delivering health and education.

These are both big books attempting to grapple with the pressing issues facing India today. They are also both rather staid books; no Poor Economics or More Than Good Intentions style human-interest stories pepper the chapters. Instead they are filled with academic references, charts and statistics to better illustrate their points. Yet, both are worth a read. They present different views on development, with areas of agreement, but also fundamental differences in approach and priorities.

If I had to pick one recommendation, it would be Bhagwati’s which seems more comprehensive, better argued and more grounded in India’s reality where the problem is often not poor policy, but poor implementation of policy. It also thankfully presents his views on policy, unlike some of his downright nasty editorials, which may be one reason the only Nobel he has won has been in Springfield.

---

[1] For ease of reading, the views are presented as Sen’s and Bhagwati’s individually, though the books are jointly written by Sen and Dreze, and Bhagwati and Panagriya respectively.

Lean in: A call for (female) guest bloggers

Andi called me out the other day for a slightly skewed gender distribution of guest bloggers. I like to think of myself as a feminist, so before I post yet another guy (tomorrow), do any female readers want to write something?

Some suggested style guidelines:

Keep it short (though up to 800 words ish)
Keep the words and sentences short
Go heavy on data, charts, and references
Go heavy on tenuous links to celebrity or scandal (think buzzfeed)

Keep it tightly focused on my narrow areas of interest and expertise (ahem, absolutely anything even vaguely related to absolutely any part of development and/or economics)

My blog reading list is also relatively light on women - suggestions there are also welcome.

How many kids attended school in South Sudan today?

You'll soon be able to find out online from live SMS reports, currently being piloted in Lainya County near Juba (1,319 girls and 1,507 boys reported present so far today in case you were wondering, 84 girls absent and 136 boys absent), with plans to roll out to the whole country. Data is reported by state, county, and even by school. As CG says, "South Sudan may not be at the top on most things, but on SMS real time school attendance monitoring, we think we may actually be leading the world." Ana Fii Inni (I am here!) is a South Sudan Ministry of Education project being supported by the DFID Girl's Education Programme.

Unrelated, I'm also told that it is possible to procure schools through the church in South Sudan for half of the $30,400 figure reported here.

How you can literally give a whole family their entire annual income today for $1,000

Planet Money & This American Life have a really great new show on GiveDirectly.

What's best are the interviews with the recipients. It makes me feel very warm to imagine someone like Bernard Omondi in a remote village waking up one morning in slight disbelief to an SMS message containing $1,000 - his family's entire annual income - more cash than he has ever had at one time in his life - and that I sent him that.

A few things really struck me.

First, that even the recipients are sceptical about cash. Although they all explained how they themselves invested the money sensibly, they were quick to judge others in the village and explain how they had totally wasted theirs on booze. Perhaps this is a psychological thing and humans just have some kind of deep suspicion that everyone else is a bit of an idiot who needs taking care of? It reminds me of the surveys that show the overwhelming majority of drivers think that they are well above average (which is impossible). It's all the other road users who are idiots. Is that right? What is the relevant psychological bias? And what is the way of addressing this? The mounting evidence doesn't really seem to be doing the trick of convincing people.

Second, and related to that point, there seemed to be some surprise that many of the households had invested in a new metal roof. A chief criticism of someone who had "wasted" the transfer was "look at him, he still has a straw roof!" Now a metal roof which lasts for ten years actually turns out to be a great investment compared to a grass roof which needs constant costly maintenance. But I also wonder whether the fact that the transfer was explicitly targeted only on people with grass roofs had anything to do with people's choices? Might some people have got the new roof because they thought that's what the transfer was for?

Third, that "we are all poor around here." There are good reasons to think that selective-targeting of transfers can damage social relations. One of the recipients says "I've lost a few friends." Which could be important if you then later need to rely on those friends when you're in a tough spot. And so GiveDirectly are trying out a new model, identify a poor village and then give cash to everyone in the village, something that people asked for repeatedly when I spoke to recipients in Kenya and Eastern DRC.

Finally, there is an interview with a woman from Heifer International, which Chris Blattman comments on here (I think he actually goes a bit lightly on them). I actually think that sending a cow could be a great intervention, but it was just an embarrassing performance, which suggested that the organisation doesn't seem to believe at all that data can be informative about effectiveness, and doesn't seem to care at all about value for money (Heifer can't or won't even provide a per family cost for their programme, because "every family is different"). My money will keep going to the organisations interested in evidence and value for money. 

"Only a parent that doesn’t love his child will send him to a public school"

This comment made by a highly educated principal of a private school shocked me greatly. This was before I interviewed public school teachers. Their responses perplexed me even more. ‘Corrupted’, ‘miscreants’, ‘thugs’, ‘wretched’, ‘from bad homes’, ‘don’t value education’ …were some of the common terms with which public school teachers described their pupils.

Fascinating stuff on low fee private schools in Lagos from Modupe Adefeso-Olateju

Hey Angus! That's why we give directly!

Angus Deaton has a new book (via Tyler), which will no doubt provide a magisterial lesson in economic history, but I'm afraid we're going to have to agree to disagree on policy.

He starts off well (there are a couple of pdf chapters online at the publishers' website here)

I believe that we—meaning those of us who are fortunate enough to have been born in the “right” countries—have a moral obligation to help reduce poverty and ill health in the world.

And then gives a rehearsal of some of those old aid arguments - yeah maybe individual projects are good for people but what about all those bad political incentives. Essentially the balance of evidence really falls on neither side, so it kind of comes down to personal judgement, or taste, or whatever, and Angus had decided that despite sending kids to school and saving lives (for sure) the negative political impacts of aid (possibly) are larger, making for a net negative total impact.

The concern with foreign aid is not about what it does to poor people around the world—indeed it touches them too rarely—but about what it does to governments in poor countries. The argument that foreign aid can make poverty worse is an argument that foreign aid makes governments less responsive to the needs of the poor, and thus does them harm.  

... Those who advocate more aid need to explain how it can be given in a way that deals with the political constraints.

Well guess what PROBLEM SOLVED Angus - you can skip the government and send yo money DIRECT TO THE POOR PEOPLE RIGHT HERE. Do try and keep up now, GiveDirectly.org have been in business for TWO YEARS already. 

Probably the best finance Minister in South Sudan yet

Congratulations and good luck to my old boss and one-time guest blogger Aggrey Tisa Sabuni on his appointment as the new finance minister of South Sudan. This is an incredibly difficult job, not likely to be helped by a possible second oil shut-down, but Aggrey Tisa is one of the smartest people I know and probably the best qualified person technically for the job that there is. I wish him well. 

New DFID Education Policy

DFID released a new "position paper" on education policy yesterday, with lots to like, and lots that chimes with how we are thinking about education at OPM.

First and foremost, the paper rightly places learning front and centre. It's really quite astonishing how neglected actual learning has been in the race to get kids into school, though as this paper shows this is starting to change.

Then there are mentions of the importance of noncognitive skills, of conceiving of education as a process of lifelong learning - including early child development, preschool, through to tertiary and technical education - and focusing on value for money*.

Then there are interesting innovations, such as Payment by results (reminiscent of CGD's "cash on delivery"), and exploring new ways of working with low cost private schools in Lagos and Sindh (OPM is working on research and evaluation for the Lagos project).

And lots of emphasis on research, including highlighting the Young Lives survey, talk of further longitudinal surveys, and a mention of the ESSPIN Nigeria survey some of my colleagues have been working on.

So what's missing?

Although the paper is right to place learning at the heart of the agenda, I don't think it really acknowledges just how little we actually know about how to improve learning. As Lant Pritchett demonstrates in his new book - if you add up all of the results from individual evaluations of various inputs, the total impact on learning outcomes just doesn't come close to getting us to where we want to be. 

This implies we need some radical experimentation. Payment by results and low cost private schools are two promising avenues, but given that this is, in DFID's words, a "learning crisis" - are these enough?

So what to do? Well firstly just more experiments, both large and small, and urgently. 

One idea would be to listen to the one thing that J-PAL and IPA are really pushing. After over a hundred large scale experiments and evaluations of education projects, they have one big idea, which happens to fit pretty neatly with DFID's agenda. This is remedial education, using young low-cost teaching assistants to help the weakest kids catch up. This is an idea developed by Indian NGO Pratham and with rigorous experimental evidence of impact, which IPA has taken and is working with the Government of Ghana to replicate and evaluate on a national scale, working through government systems. Why isn't DFID funding the next programme like this?

---

*As a sad little footnote, unsurprisingly DFID's worst value for money in education seems to be coming from, you guessed it;

"The cost of classroom construction, for example, ranges from $1,400 in Ethiopia to $30,400 in South Sudan, where years of conflict have dramatically increased the cost of materials and the mobility of skilled personnel.

What would Einstein say about schooling for all?

From a letter to his 11-year old son 

I am very pleased that you find joy with the piano. This and carpentry are in my opinion for your age the best pursuits, better even than school. Because those are things which fit a young person such as you very well. Mainly play the things on the piano which please you, even if the teacher does not assign those. That is the way to learn the most, that when you are doing something with such enjoyment that you don’t notice that the time passes.

How to make maps

It's not new, but thought I'd share this handy tool "StatPlanet" for mapping country-level and sub-national state-level data. It took me half an hour to download and figure out, and then all you have to do is import your Excel spreadsheet in the right format, and it spits out pretty maps. The list of countries with sub-national maps built-in is here. The interactive flash maps are pretty cool too. 

Nigeria: Primary School Net Attendance Ratio (%)

Source: Nigeria Education Data Survey 2010

Evidence-based policy-making US-style

Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.

...

Since 1990, the federal government has put 11 large social programs, collectively costing taxpayers more than $10 billion a year, through randomized controlled trials, the gold standard of evaluation. Ten out of the 11—including Upward Bound and Job Corps—showed “weak or no positive effects”

Just in case you thought that there was any danger of the whole results agenda and RCT-fetishism taking over in American politics. From an excellent piece in last month's The Atlantic, which by the way is generally fantastic, I just bought a paper copy for the first time and the whole thing was full of interesting - a critical look at the evidence on over 35 female fertility, a thing about how much health food is actually really unhealthy, a note about how recycling can actually increase carbon emissions because it needs more trucks on the streets, and a piece discussing relationships and gender politics and family from the perspective of a man who has sacrificed his career for his wife's. 

The one thing that does get delivered in India's government schools

Excellent writing by Abhijeet in the Guardian providing some perspective on the recent deaths in Bihar from school meals.

the midday meals, which reach about 120 million children on every school day, are probably the most successful of all interventions in education that the Indian state has delivered in the past decade. On any school day, a quarter of teachers are absent from government schools (pdf), only 45% of those in school are teaching, but in 87% of schools, a hot meal is served (pdf).

The revolution might not be televised, but what about development?

One of the latest big things in education research is the importance of what economists have labelled "non-cognitive skills," psychologists call "psychosocial competencies", and humans have been known to call "character." Various studies have found that "cognitive skills" - literacy, numeracy, etc, actually don't do a fantastic job of explaining success later in life, and that it is a bunch of other soft skills which really do matter - things like perseverance, curiosity, conscientiousness, optimism, and self-control (perhaps someone should tell Michael Gove?).

All of this was the subject of Stefan Dercon's plenary at Young Lives last week (slides here). The good news is that these characteristics appear to be malleable even into adulthood.

Apparently there is one question to measure "locus of control" which correlates highly with aspirations and investing in the future. The question is simple - which of the following two statements do you most agree with:

1. “Each person is primarily responsible for his/her success or failure in life.”
2. “One’s success or failure in life is a matter of his/her destiny/fate.”

If how you think about this question determines how much you save and invest for the future and your attitudes towards this can be positively shifted, then maybe there is a "fatalism trap" which intervention can break. Stefan presented results from an experiment run in Ethiopia to see if optimism could be cultivated by showing poor remote villagers mini-documentaries of inspirational success stories on TV. The shows told the stories of local people who had worked their way out of poverty. The experiment also included both a regular control group and a "placebo" regular TV show, without the particular success stories.

Six months later - the villagers were still talking excitedly talking about the TV show, and more importantly actually spending more on investing in education for their kids.

Of course, the tragic danger is whether these newly raised aspirations will turn out to be a false hope if the economic opportunities to make these investments pay off aren't available.

Yet another reason why TV will save the world?