Paul Krugman, Noah Smith, and Bryan Caplan had an interesting debate last week on the use (and misuse) of maths in economics.
A helpful illustration is provided this month by a new paper by Lant Pritchett and Justin Sandefur on external validity and RCTs (handy Charles Kenny summary in BusinessWeek here).
The concept of external validity is pretty simple to grasp intuitively. An experiment might give you a good estimate of the impact of a programme in a certain context, but it can't tell you if the same programme will have the same impact in a totally different context.
This is something which is especially obvious when you are actually working on national policy. When you are a writing a brief for a politician or an NGO on an issue, it would just feel stupid to lead with evidence from a totally different country if there is any data at all from the country you are actually in. Not that studies from other countries are uninteresting, but it is just blindingly clear that there are a lot of differences in political, social, and economic context between countries that might make results from a similar programme quite different, and so to use some caution in drawing conclusions, even from perfectly executed experimental studies.
At yet at the same time, "external validity" can seem like a bit of a hand-wavy rebuttal compared to all of the extensive technical theory around internal validity - whether your study is likely to give a biased estimate of programme impact on the population you are studying. There is a lot of detailed methodological analysis looking at exactly what the causes of bias are in different studies and how this bias can be best avoided. So what Pritchett's and Sandefur's paper does is add some detail to the our understanding of external validity, add some maths, and somehow make the critique seem somehow weightier. I think I still find their empirical examples more compelling than their theory, because I'm slow with maths and more interested in empirics, but nonetheless it does seem important to have that kind of systematic logical thinking through of the detail of a problem.
- Economath - not totally useless, but you can probably get the intuition without it.
- External validity - an important concern, and sometimes contextual understanding matters more than clean identification - but also a reason for more experiments where possible not less
No comments:
Post a Comment