The path to scale: From randomised control trial to scalable programme

Article

Published 14.11.17
Photo credit:
Stephanie Skinner (Evidence Action).

An RCT conducted in Bangladesh found positive impacts on incomes and calorie consumption; academics and practitioners work together to scale it

The use of randomised control trials (RCTs) to evaluate development programmes has become ubiquitous in development economics research. Academic economists conduct impact evaluations not only to evaluate the specific pilot programme we study, but to attempt to inform the bigger policy question: How well would this programme fare if it was scaled up district-wide, state-wide or country-wide by the national government, donors, or a large non-governmental organisation (NGO)?

This naturally leads to an interest in understanding the ‘scale-up potential’ of development programmes, and raises the question: How do we go from results from a tightly controlled experiment to an intervention that can actually improve the lives of millions of people? This two-part column reflects on our experience over the last several years transforming an RCT that offered small loans for seasonal migration in Northern Bangladesh into a programme called No Lean Season, which is now on a path towards reaching hundreds of thousands of poor households in the region, and thousands more in other countries. In the first of these two columns, we detail the information we learned from the original RCT and how this fed into the programme design for No Lean Season.

Scaling up comes with new challenges

There are numerous practical and political challenges to go from RCT to a scaled-up programme.  For instance, a research outfit that can deliver an intervention to 2,000 people is unlikely to have the capacity – or be the best partner – to implement a programme for 200,000 or 2 million people.  Local and country politics also come to the surface very quickly when programme target numbers increase, requiring advocacy and communications skills to build the requisite buy-in from political and regulatory authorities. Logistical and programmatic expertise are also needed to design and adapt systematic implementation protocols that can be delivered at scale reliably and efficiently.

Aside from these implementation challenges are the economic questions that Deaton and Cartwright (2016) label collectively the ‘transportation problem’ – how to use the results from the original RCT to think about outcomes in a different (or larger) context. This encompasses the need to not just account for general equilibrium effects that may arise at scale, but also explore additional effects beyond the ones originally measured and that may be more relevant as the balance of programme goals shift slightly from research and knowledge towards impact and cost-effectiveness.

No Lean Season is set up as an academic–policy partnership between Evidence Action, a non-profit organisation that specializes in scaling evidence-based programmes, and an academic economist based at Yale University working with a large research team to address precisely these issues. Tackling these questions requires not only careful thought, utilising economic theory and corresponding data analysis, but also the ability to implement and pressure-test scaled-up versions of the programme and solve practical problems as they arise in the field.

The starting point: Compelling results

In 2008-2011, one of us was part of a research team that conducted an RCT on 1,900 households to test whether providing small loans or grants for the rural poor in Northern Bangladesh increased their seasonal migration and, in turn, improved consumption and food security during a very vulnerable time of the year. The results, published in Econometrica in 2014 (Bryan et al. 2014), showed that small loans and grants worth about US$9 (the cost of a round-trip bus ticket and a few days of food), and disbursed conditionally on migration, increased domestic temporary migration rates from 36% to 58% during the lean season.

There was no meaningful difference in effects between a no-interest loan and a grant, and this 22 percentage point jump in migration translated into welfare improvements as well. Food and non-food expenditures among beneficiary households increased by 30-35%, and calorie intake rose by 550-700 per person per day, equivalent to a full meal during a time when meals are regularly skipped. Importantly, targeted households continued to migrate at a rate 8-10 percentage points higher than that of the control group one and three years later.

These initial RCT results pointed to a treatment with great promise, and drew the attention of Evidence Action, given its commitment to testing and bringing to scale evidence-based interventions that have the potential to benefit millions of people. This seasonal migration subsidy is relatively cost-effective (Hollander 2017) – showing large welfare impacts on targeted households, while remaining very inexpensive on a per-person basis – and the logistics of expanding its reach within the country seemed possible with a partner (RDRS Bangladesh) that not only has extensive experience in region but had also participated in the original RCT.

Looking beyond income and consumption

In the world of RCT-based research, an intervention can be hailed as promising if it shows positive economic effects, such as improvements in consumption or even caloric intake. But when an NGO or government is looking into funding and implementing a much larger version of this RCT, targeting thousands or millions of people, there is more at stake. In particular, it is important to know whether this same intervention might also come with negative collateral effects, on both the subjects and other households. As Deaton and Cartwright (2016) warn in a general critique of the limitations of RCTs for informing policies, “just because an intervention works, and because the investigator thinks the intervention makes people better off, is no guarantee that it actually does so”.                     

The non-economic welfare effects of No Lean Season

In our work to move No Lean Season from an RCT to a programme, we have carefully considered and made efforts to test both general equilibrium effects and potential non-economic effects. For the latter, we have thus far explored three areas.

1) The effect of seasonal migration on risk-sharing networks

As poor households are inter-linked through informal insurance networks, encouraging the seasonal migration of some members may alter the risk-sharing options available to migrants and non-migrants.

In theory, migration may improve the diversification of income, decreasing risk for migrants as well as others in their network. On the other hand, as migrants improve their own risk portfolio, they may choose to leave their network, rendering the remaining members worse off (as they have fewer people with whom to share the risk). Imperfect information sharing from the migrant to others in the network may also create a moral hazard, as migrants take on more risk in response to weaker monitoring while they are away.

The balance between these three forces in a given context can only be determined empirically. Preliminary findings reveal that seasonal migration in Northern Bangladesh benefits migrants along with others in their networks, as it improves risk-sharing and smooths consumption for the entire network.

2) The impact of migration on gender and social norms and political beliefs

Here, we ask whether seasonal migration leads to social changes as migrants are exposed to urban lifestyles as well as perhaps acquiring a more acute knowledge of differences in income and living conditions between their villages and destination areas. Targeted households belong to the bottom economic half in their village, but with migration they may learn that they are in fact in the bottom quantile or quintile when compared to urban areas, creating tension, discontent about inequality, and perhaps political unrest. This may be of concern to local and regional policymakers as scale-up is considered.

A related question is whether seasonal migration changes gender norms, as migrants tend to be exclusively male, and their partners are left behind with additional responsibilities during the season. Interestingly, early results in this area indicate no effects of seasonal migration on gender norms, social attitudes, or political beliefs. We speculate that this may be at least partially explained by a high baseline migration rate, so that rural households in this context are already (directly or indirectly) exposed to urban ideas and have knowledge about urban ideas, conditions, and inequality.

3) The welfare of migrants and their families during migration

We are motivated by the original finding that, one and three years after the incentive, migration remains higher among targeted households relative to the control group but nonetheless decreases relative to the intervention year. In this area, we explore whether there is a disutility to migration, as only a subset of incentivised migrants choose to re-migrate on their own once a loan is not offered.

The findings here are mixed. On the one hand, migration does impose some non-monetary costs as migrants, if they can help it, prefer to live at home with their families rather than live in poor urban conditions. On the other hand, members of poor rural households that have faced a recent spate of bad shocks in the village really do need to a better labour market temporarily to insure themselves against the attendant adverse consequences.  Providing migration opportunities is extremely valuable in such situations. The travel cost acts as a constraint exactly in those periods, when those households have been forced to draw down their savings.

In this way, offering seasonal migration loans is more cost-effective than an unconditional cash transfer – recently gaining traction as the ‘default comparison’ in development – as it better targets poor and vulnerable households. Only those who need to migrate will do so, and for those households, welfare improves during the lean season.

Moving along on the path to scale

These results give us, and Evidence Action, the confidence to explore the next steps in scaling the intervention. In line with the original study results, No Lean Season offers small no-interest loans to support seasonal migration for poor, rural labourers in Northern Bangladesh, and is projected to directly impact over 40,000 households this year.

Editor’s Note: This article is based on an IGC project

Photo credit: Stephanie Skinner (Evidence Action).

References

Akram, A and M Mobarak (2016), "Effects of Emigration on Rural Labor Markets", NBER Working Paper 23929.

Bryan, G, S Chowdhury and A M Mobarak (2014), "Underinvestment in a profitable technology: The case of seasonal migration in Bangladesh", Econometrica 82(5).

Deaton, A and N Cartwright (2016), "The limitations of randomised controlled trials", VoxEU.org, 9 November.

Hollander, C (2017), "Why we’re considering No Lean Season as a potential 2017 top charity", The GiveWell Blog.