A research agenda on education & institutions

From Tessa Bold & Jakob Svensson for the DFID-OPM-Paris School of Economics research programme "EDI"

 

1. A focus on learning in primary is still essential - don’t get too distracted by secondary and tertiary

2. More focus on teachers’ effort, knowledge, and skills

3. How do we go from pilots to scaled-up programs? (and related - can we design interventions that explicitly allow for existing implementation constraints at scale)

4. How can we use ICT to bring down the cost of sharing information on performance?

5. More research on public-private partnerships such as voucher programs

Stop highlighting our differences? #moreincommon

Last night at my local primary school Governor meeting one of the other governors objected to a table showing a disaggregation of recent pupil discipline issues categorised by ethnic grouping. “Should we really be calling children ‘White Other’ or ‘Black Other’?” Turns out these are the standard official government categories offered to students/parents to self-identify with. As a researcher I'm naturally interested in as many descriptive categories as possible to help understand the factors that drive differences in outcomes between individuals, but every time we ask the question we also ask people to think in ethnic or racial or national terms, highlighting our differences not the more in common. 

 

As Chris Dillow wrote recently in an excellent take-down of David Goodheart:

"The thing is, we all have multiple identities: I’m tall, white, Oxford-educated, bald, heterosexual, male, bourgeois with a working class background, an economist, an atheist with a Methodist upbringing. And so on and on. The question is not: what are my identities? But rather: which of these identities matter?

 

… 

 

Even if you accept biological essentialism, the question of which of our multiple identities becomes salient is surely in large part a social construct.

 

...

 

No good can come from raising the salience of racial or ethnic identities."

The issue comes up often in national censuses. For the last South Sudan census in 2008 it was decided it was too politically charged to ask people their ethnic group. Lebanon hasn’t had a census to count the number of Christians and Muslims since 1932. In the UK, the government recently started asking schools for pupil’s nationalities, with a stated aim of allowing for better targeting of support, but leading to widespread suspicion and calls for a boycott. 

 

The first step to thinking about when we should and should not ask for ethnic identities might be assigning some plausible values to the likely costs and benefits of doing so. A new paper by Evan S. Lieberman and Prerna Singh has taken a systematic approach and coded over 1000 census questionnaires for 150 countries over 200 years and whether they ask for ethnic identities. 

"Through a series of cross-national statistical analyses, the authors find a robust association between enumeration of ethnic cleavages on the census and various forms of competition and conflict, including violent ethnic civil war”.

That seems like a pretty high price to pay.

The Political Economy of Public Sector Performance Management Reform

Reflections from Prajapati Trivedi, founding Secretary of the Performance Management Division in the Government of India Cabinet Secretariat, in Governance. 

"The new government of Prime Minister Modi never formally declared that it is closing the RFD system. It simply stopped asking the departments to prepare RFDs (performance agreements). Indeed, the government went on to appoint three more Secretaries for Performance Management as my successors. The system was, however, allowed to atrophy and no formal answer about the status of the RFD system was either given in the Parliament in response to questions on the topic or was forthcoming under India's Right to Information (RTI) act. Thus, we can only speculate why the RFD system came to an abrupt end.

First, it is possible that the review of the working of the RFD system in the past 4 years by the incoming Modi government revealed a story that did not match their election rhetoric. Modi had portrayed the outgoing government of Singh as weak on governance and could not, therefore, acknowledge the existence of a rigorous system of performance monitoring and evaluation of government departments. After all, it had promised to do in its election manifesto what was already being done.

Second, the review of actual results for the performance of individual government departments perhaps revealed that “reality” of performance was better than the “perception.” It is fair to say that Manmohan Singh lost elections because he could not “talk the walk.” The performance data revealed that on average government departments were achieving 80% of the targets assigned to them via performance agreement (RFDs). By contrast, the opinion polls at the time revealed that the electorate rated the government performance at around only forty percent 40%. Thus, continuing the RFD system would have revealed facts that went against the main narrative on which the government of Modi came to power.

Third, it is possible that the new government found that the performance results for the past 4 years based on the departments' ability to meet agreed commitments did not meet their preconceived biases.

Fourth, a system based on ex ante agreements and objective evaluation of performance at the end of the year reduces discretion and was perhaps seen as inimical to the personalized style of management preferred by the incoming Prime Minister.

And fifth, I have yet to come across any workplace where performance evaluation is welcomed by staff. Senior career civil servants did feel the pressure to perform and were waiting to sow seeds of doubt in the minds of incoming administration. Performance management is a leadership issue and not a popular issue.

There are several key lessons of my experience that may be relevant for policy makers working on a similar system in their own countries.

We succeeded beyond our wildest expectations in terms of the scope and coverage of the performance management policy because we emphasized simplicity over complexity. We defined performance management as simply the ability of the department to deliver what it had promised.

It is my strong conviction that unless you can reduce the performance measurement to a score, it will remain difficult to grasp and hence difficult to sustain over time. For the performance score to be meaningful, however, performance management must be based on ex ante commitments and must cover all aspects of departmental operations.

Performance management is best implemented as a big-bang effort. Pilots in performance management do not survive because those departments chosen as pilots feel they are being singled out for political reasons.

Finally, the single biggest mistake of the outgoing government was to not enshrine the RFD policy in a law. A similar policy for accountability of state-owned enterprises in India was embedded in a law in 1991 and it has survived changes in government." 

Liberia Fact of the Day

"The treaties that govern space allow private individuals and corporations to travel the stars, but only with the licensure and legal backing of an earthbound government. It’s similar that way to the laws of the sea. And today, on Earth’s oceans, more than 11 percent of all the tons of freight shipped is carried on boats that fly the Liberian flag (In contrast, U.S.-registered ships carry just 0.7 percent of the freight tonnage).

In exchange for lower taxes and looser regulations, both the shipping companies of the present and the Martian explorers of tomorrow could pay to register their vessel with a small country they have no other connection to (Liberia earns more than $20 million a year this way) and carry its flag (and laws) with them, wherever they go."

Maggie Koerth-Baker at 538 (via The Browser)

The key to better education systems is accountability. So how on earth do we do that?

And what do we even actually mean when we talk about accountability?

Perhaps the key theme emerging from research on reforming education systems is accountability. But accountability means different things to different people. To start with, many think first of bottom-up (‘citizen’ or ‘social’) accountability. But increasingly in development economics, enthusiasm is waning for bottom-up social accountability as studies show limited impacts on outcomes. The implicit conclusion then is to revisit top-down (state) accountability. As Rachel Glennerster (Executive Director of J-PAL) wrote recently: 

"For years the Bank and other international agencies have sought to give the poor a voice in health, education, and infrastructure decisions through channels unrelated to politics. They have set up school committees, clinic committees, water and sanitation committees on which sit members of the local community. These members are then asked to “oversee” the work of teachers, health workers, and others. But a body of research suggests that this approach has produced disappointing results."

One striking example of this kind of research is Ben Olken’s work on infrastructure in Indonesia, which directly compared the effect of a top-down audit (which was effective) with bottom-up community monitoring (ineffective).

So what do we mean by top-down accountability for schools?

Within top-down accountability there are a range of methods by which schools and teachers could be held accountable for their performance. Three broad types stand out:

• Student test scores (whether simple averages or more sophisticated value-added models)
• Professional judgement (e.g. based on lesson observations)
• Student feedback

The Gates Foundation published a major report in 2013 on how to “Measure Effective Teaching”, concluding that each of these three types of measurement has strengths and weaknesses, and that the best teacher evaluation system should therefore combine all three: test scores, lesson observations, and student feedback.

By contrast, when it comes to holding head teachers accountable for school performance, the focus in both US policy reform and research is almost entirely on test scores. There are good reasons for this - education in the US has developed as a fundamentally local activity built on bottom up accountability, often with small and relatively autonomous school districts, with little tradition of supervision by higher levels of government. Nevertheless, as Helen Ladd, a Professor of Public Policy and Economics at Duke University and an expert in school accountability, wrote on the Brookings blog last year:

"The current test based approach to accountability is far too narrow … has led to many unintended and negative consequences. It has narrowed the curriculum, induced schools and teachers to focus on what is being tested, led to teaching to the test, induced schools to manipulate the testing pool, and in some well-publicized cases induced some school teachers and administrators to cheat. 

Now is the time to experiment with inspections for school accountability … 

Such systems have been used extensively in other countries … provide useful information to schools … disseminate information on best practices … draw attention to school activities that have the potential to generate a broader range of educational outcomes than just performance on test scores … [and] treats schools fairly by holding them accountable only for the practices under their control … 

The few studies that have focused on the single narrow measure of student test scores have found small positive effects."

A report by the US think tank “Education Sector” also highlights the value of feedback provided through inspection systems to schools.

"Like many of its American counterparts, Peterhouse Primary School in Norfolk County, England, received some bad news early in 2010. Peterhouse had failed to pass muster under its government’s school accountability scheme, and it would need to take special measures to improve. But that is where the similarity ended. As Peterhouse’s leaders worked to develop an action plan for improving, they benefited from a resource few, if any, American schools enjoy. Bundled right along with the school’s accountability rating came a 14-page narrative report on the school’s specific strengths and weaknesses in key areas, such as leadership and classroom teaching, along with a list of top-priority recommendations for tackling problems. With the report in hand, Peterhouse improved rapidly, taking only 14 months to boost its rating substantially."

In the UK, ‘Ofsted’ reports are based on a composite of several different dimensions, including test scores, but also as importantly, independent assessments of school leadership, teaching practices and support for vulnerable students.

There is a huge lack of evidence on school accountability

This blind spot on school inspections isn’t just a problem for education in the US, though. The US is also home to most of the leading researchers on education in developing countries, and that research agenda is skewed by the US policy and research context. The leading education economists don’t study inspections because there aren’t any in the places they live.

The best literature reviews in economics can often be found in the “Handbook of Economics” series and the Journal of Economic Perspectives (JEP). The Handbook article on "School Accountability" from 2011 exclusively discusses the kind of test-based accountability that is common in the US, with no mention of the kind of inspections common in Europe and other countries at all. A recent JEP symposium on Schools and Accountability includes a great article by Isaac Mbiti, a Research on Improving Systems of Education (RISE) researcher, on ’The Need for Accountability in Education in Developing Countries” which includes; however, only one paragraph on school inspections. Another great resource on this topic is the 2011 World Bank book, "Making Schools Work: New Evidence on Accountability Reforms”. This 'must-read' 250-page book has only two paragraphs on school inspections.

This is in part a disciplinary point - it is mostly a blind-spot of economists. School inspections have been studied in more detail by education researchers. But economists have genuinely raised the bar in terms of using rigorous quantitative methods to study education. In total, I count 7 causal studies of the effects of inspections on learning outcomes - 3 by economists and 4 by education researchers.

Putting aside learning outcomes for a moment, one study from leading RISE researchers, Karthik Muralidharan and Jishnu Das (with Alaka Holla and Aakash Mohpal), in rural India finds that “increases in the frequency of inspections are strongly correlated with lower teacher absence”, which could be expected to lead to more learning as a result. However, no such correlation was found for other countries in a companion study (Bangladesh, Ecuador, Indonesia, Peru, and Uganda).

There is also fascinating qualitative work by fellow RISE researcher, Yamini Aiyar (Director of the ‘Accountability Initiative’ and collaborator of RISE researchers Rukmini Banerji, Karthik Muralidharan, and Lant Pritchett) and co-authors, that looks into how local level education administrators view their role in the Indian state of Bihar. The most frequently used term by local officials to describe their role was a “Post Officer” - someone who simply passes messages up and down the bureaucratic chain - “a powerless cog in a large machine with little authority to take decisions." A survey of their time use found that on average a school visit lasts around one hour, with 15 minutes of that time spent in a classroom, with the rest spent “checking attendance registers, examining the mid-day meal scheme and engaging in casual conversations with headmasters and teacher colleagues … the process of school visits was reduced to a mechanical exercise of ticking boxes and collecting relevant data. Academic 'mentoring' of teachers was not part of the agenda.”

At the Education Partnerships Group (EPG) and RISE we’re hoping to help fill this policy and research gap, through nascent school evaluation reforms supported by EPG in Madhya Pradesh, India, that will be studied by the RISE India research team, and an ongoing reform project working with the government of the Western Cape in South Africa. Everything we know about education systems in developing countries suggests that they are in crisis, and that a key part of the solution is around accountability. Yet we know little about how school inspections - the main component of school accountability in most developed countries - might be more effective in poor countries. It’s time we changed that.

This post appeared first on the RISE website. 

Rising DFID Spending hasn't Crowded Out Private Giving

Last week I was poking around the ESRC’s 'Administrative Data Research Network’ and discovered the Charity Commission data download website - containing every annual financial return made by every individual charity in England and Wales since 2007. The data comes in a slightly weird file format that I’d never heard of, but thankfully the NCVO have a very helpful guide and Python code for converting the data into .csv format (which was easy enough to use that I managed to figure out how to run without ever having really used Python). 

One obvious question you could ask with this data is whether the private income of international charities has dropped as DFID spending has gone up (more than doubled over the same period) - it is conceivable that people might decide that they could give less to international charity as more of their tax money is being distributed by DFID.

That does not seem to be the case at all. There are two ways of identifying international charities - by their stated area of operation, or by their stated objective category. I’ve coded charities that have no UK activities as “International”, and also picked out the charities that ticked the box for "Overseas Aid/Famine Relief” as their activity category. These two categories do overlap but far from perfectly. 

Charities have multiple categories of income - I focus here on the ‘voluntary’ category which basically means all donations, whether large or small. 

Charities with exclusively international activities, and those focused on 'overseas aid' did appear to take more of a hit than domestic charities from the 2008 global financial crisis and recession, but since then growth has tracked the income of other charities (and is 40-50% higher in 2015 than in 2007 (not adjusting for inflation)). 

You can download the Stata code here, csv files (large) here, and variable descriptions here.

Introducing... the Global Schools Forum

There’s nothing like sitting in a room full of people who build and run schools in the developing world to make you feel pretty inadequate. At least I did, last week at the Global Schools Forum. It can feel like a pretty long and abstract chain from the kind of policy research and evaluation that I do through to better policies and better outcomes, and I envy being able to see directly a tangible difference for real people.

You may have heard of the emergence of some international low-cost private school chains such as Bridge International Academies, but the movement is growing quickly, and there are many new organisations trying to do similar things that you probably haven’t heard of - some profit-making, some non-profit, some that charge fees, some that don’t, international, local, big, small, and everything in between. The biggest school operator you don’t hear that much about is the Bangladeshi NGO BRAC, who run thousands and thousands of fee-free schools.

Last week a whole range of school operators and the donors who support them gathered at the 2nd Annual Meeting of the “Global Schools Forum” (GSF); a new membership organisation of 26 school networks (of which 14 for profit and 12 non-profit) operating in 25 countries, and 17 donors and financing organisations, with networks ranging from 1 to 48,000 schools. Running one school is hard enough; trying to disrupt a dysfunctional system by growing a chain of schools is harder. One of the goals of the GSF is to help create a community of practice for school operators, and a “How-to Guide” for scale, sharing information about the best IT and finance systems, the best assessments for tracking performance, or the best training consultants. It’s a place to connect operators with new people and new ideas.

This year we heard more about public-private partnerships than last year, in part because of the presence of several of the operators in the Partnership Schools for Liberia (PSL) pilot. Government schools (with government teachers) will be managed by a range of local and international providers, including the Liberian Stella Maris Polytechnic and LIYONET, international NGOs (Street Child, BRAC, More Than Me) and school chains (Bridge International Academies, Rising Academies, Omega Schools). Other operators at the forum came from India (Gyanshala, Seed Schools, Sodha Schools), South Africa (SPARK, Streetlight Schools, African School for Excellence, Nova Pioneer), and East Africa (Peas, Silverleaf Academies, Kidogo, Scholé), to name just a few.

So what?

The number of non-state school operators planning for scale is rapidly increasing, but even at dramatic rates of growth, it would take a long time to reach any kind of significant proportion of schools. There are two possible routes to scale - either growing chains and networks to scale themselves, and/or acting as demonstration projects for government, to prove what is possible. This second route was highlighted by a number of speakers. Anecdotally at least, many exciting school reforms seem to come from the personal experience of government Ministers actually seeing something better in practice with their own eyes. More rigorously, Roland Fryer has demonstrated in the US with a randomized experiment that it is possible to “inject charter school best practices into public schools” and achieve positive gains in learning.

To find out more about the Global Schools Forum keep an eye on the website (coming soon) globalschoolsforum.org and follow @GSF_talks on twitter.

The Global Schools Forum is supported by The Education Partnerships Group (EPG) @EPG_Edu, UBS Optimus Foundation @UBSOptimus, Pearson Affordable Learning Fund @AffordableLearn, and Omidyar Network @OmidyarNetwork.