Monitoring a programme by calling recipients is a cost-effective way to improve implementation quality
An age-old problem faced by leaders is learning what is actually happening on ‘the ground’. Many stories describe historical rulers disguising themselves to roam around their kingdoms and observe the true state of affairs. In these accounts, this approach motivated public servants to perform better, as they never knew if their monarch may be watching, and the rulers used the information they collected to make better policy decisions.
Observing the quality of public programme implementation remains a core governance challenge for policymakers today. In most contexts, there is a lack of up-to-date and representative data to provide an objective picture of ground realities. Even digitised administrative data often lacks information on key outcomes, and lower-level officials have incentives to underreport problems, hide corruption, and exaggerate their performance (Singh 2019). This makes public sector management an incredibly challenging problem, since like any organisation, a government can only manage its personnel to the extent that it can measure their performance (Bloom and Van Reenen 2007).
Leveraging widespread mobile coverage to observe policy implementation
Technological advances may make it possible for modern policymakers to do better than traveling in disguise. Mobile-phone penetration in low-income countries has grown, from 1 mobile subscriber per 100 people in 2002, to 62 in 2017 (World Bank 2018). Given that most households are now reachable by phone, one option is for governments to proactively call representative samples of citizens to collect information. This can provide real-time data on administration of public programmes and provide performance measurement that allows better management of frontline bureaucrats.
Governments have begun to use outbound phone calls to gather data on public service delivery, with Pakistan's Citizen Feedback Monitoring Programme and Andhra Pradesh's Real Time Governance initiative as prominent examples (Masud 2015; World Bank 2016; Gelb 2019). However, there is only limited quantitative evidence to date on the impact of phone-based monitoring.
Monitoring distribution of money for agricultural inputs
In Muralidharan et al. (2021), we tested how the presence of phone-based monitoring can affect bureaucratic performance in the Indian state of Telangana. This was carried out in the context of a high-profile government cash transfer programme (Rythu Bandhu Scheme, or RBS), in which the state government provides farmers with Rs. 4,000/acre (US$55) per agricultural season to help pay for key inputs like seeds and fertilizer.
In 2018, the government distributed $1.8 billion, equal to 7% of the state's annual budget, under this scheme. Payment was issued in the form of cheques, distributed to farmers by local agricultural officials. The programme was a top priority for the state government, and was carefully monitored. Yet – as usual – there were various implementation risks, including cheques not issued, not delivered, delivered late, or delivered with a request for a bribe.
Study methodology
We implemented an experimental, at-scale test of phone-based performance measurement. Through a randomised assignment process, a quarter of the 498 sub-districts in the state were assigned to the treatment group. In these sub-districts, the implementing officials were told that a call center would:
- call at least 100 of the farmers in their sub-district,
- collect information on implementation outcomes (including whether and when farmers received their cheques, and corruption during the process), and
- produce reports visible to them and their supervisors.
To evaluate the impact of phone-based monitoring, we used administrative bank records of whether the cheques were cashed, and the exact date of encashment. This high-quality administrative data on the entire universe of 5.7 million potential programme beneficiaries permits precise estimation of the impact of the phone-based monitoring treatment on the entire population – including both, beneficiaries with and without cell phones.
Phone-based monitoring made the scheme work better for all farmers, including those without mobile phones
The presence of the call-center had a significant positive impact on implementation. Even in areas without phone-based monitoring, implementation was reasonably good: 83% of farmers received and cashed their cheques. However, farmers did better in areas with phone-based monitoring:
- They were 1.3 percentage points more likely to receive and encash their cheques; expressed differently, the intervention led to a 7.8% reduction in the number of farmers who did not receive their benefits.
- The monitoring benefited the disadvantaged even more, with farmers in the bottom quarter of land holdings becoming 2.2 percentage points more likely to encash their cheques – about twice the effect observed for the treatment group as a whole.
- Farmers in treated areas were also 2.4 percentage points more likely to receive their transfer ‘on-time’ (that is, prior to the arrival of the monsoon, which set in on June 8th, 2018).
- Benefits were similar for farmers with and without phones, suggesting that officials did not cater only to those with a phone.
These effects correspond to a $3.9 million increase in transfers that were delivered on-time, a $1 million increase in the total amount delivered, and 17,771 additional farmers encashing their cheques. If phone-based monitoring had been extended to the entire state over both agricultural seasons, these estimates suggest that $33.1 million more would have been delivered on time, an additional $8.6 million would ultimately have been delivered, and around 155,000 additional farmers would have received their payments.
Phone call-based monitoring is a cost-effective and versatile technique
Overall, the call centers were highly cost-effective, as the incremental cost per additional dollar of benefits ever delivered to beneficiaries was 3.6 cents – a lower administrative cost than nearly any other anti-poverty programme for which data is available.
Empirical evidence on service delivery in developing countries suggests that increased top-down administrative monitoring can be an effective tool for improving last-mile service delivery (Olken 2007). However, a practical barrier to the use of top-down monitoring has been the challenge of obtaining credible, high-frequency data on last-mile service delivery, at a sufficiently spatially disaggregated level to hold appropriate officials accountable.
Other recent papers have tested specialised approaches for using technology to improve governance, such as monitoring worker attendance with time clocks (Banerjee et al. 2008; Dhaliwal et al. 2017) or custom smart-phone applications (Callen et al. 2018). Relative to these specialised approaches, measurement by phone call has an ability to scale across a wide range of places, programmes and outcomes. This type of system can measure anything that can be observed and then described over the phone, ranging from the presence of teachers in schools to corruption in benefit delivery. This type of system is also low-cost, fast to set up, and can respond quickly to changing circumstances on the ground.
Implications for policy: From performance management to accountability and participation
Other programmes have significantly more room for improvement than the studied programme in Telangana, suggesting the benefits of this approach could be higher in other settings; for example, Aker and Ksoll (2018) find significant benefits of phone-based monitoring intervention in the context of an adult education programme in Niger.
Monitoring by phone could also be paired with explicit rewards or sanctions to motivate implementing officials (something not done in our study), and with public disclosure of statistics to build bottom-up democratic accountability. More broadly, regularly measuring performance can give managers objective metrics to guide personnel management, which global evidence suggests is the most important component of management (Bloom and Van Reenen, 2010).
Incorporating phone-based data collection into policymaking has potential beyond performance monitoring. One possibility is that policy makers could use these systems to collect data that help them in making decisions. For example, if designing a targeted cash transfer programme, a policy maker might use the same call center infrastructure to gather data on beneficiary financial literacy and use of formal banking systems, to better target enrollment.
While further experimentation with phone-based data collection by governments can help tweak this approach and test other use cases, existing evidence suggests that doing so can have large returns.
References
Aker, J and C Ksoll (2018), Call me educated: Evidence from a mobile monitoring experiment in Niger, Working Paper 406, Centre for Global Development.
Banerjee, A, E Duflo, and R Glennerster (2010), “Putting a band-aid on a corpse: Incentives for nurses in the Indian public health care system”, Journal of the European Economic Association 6 (2-3): 487–500.
Bloom, N and J van Reenen (2007), “Measuring and explaining management practices across firms and countries”, The Quarterly Journal of Economics 122 (4): 1351–1408.
Bloom, N and J van Reenen (2010), “Why Do Management Practices Differ across Firms and Countries?”, Journal of Economic Perspectives 24 (1): 203–224.
Callen, M, S Gulzar, A Hasanain, M Y Khan, and A Rezaee (2018), "Data and policy decisions: Experimental evidence from Pakistan", Working Paper No. 1022, Stanford Institute of Economic Policy Research (SIEPR).
Dhaliwal, I and R Hanna (2017), “The devil is in the details: The successes and limitations of bureaucratic reform in India”, Journal of Development Economics 124: 1–21.
Gelb, A, N Mittal, and A Mukherjee (2019), "Towards real-time governance: Using digital feedback to improve service, voice, and accountability", Centre for Global Development.
Masud, M O (2015), Calling the Public to Empower the State: Pakistan’s Citizen Feedback Monitoring Programme (2008-2014), Technical Report, Princeton University Innovations for Successful Societies.
Muralidharan, K, P Niehaus, S Sukhtankar, and J Weaver (2021), “Improving last-mile service delivery using phone-based monitoring”, American Economic Journal: Applied Economics 13.2: 52-82.
Olken, B (2007), “Monitoring corruption: Evidence from a field experiment in Indonesia”, Journal of Political Economy 115 (2): 200–249.
Singh, A (2019), "The myths of official measurement: Auditing and improving education data in developing countries", Research on Improving Systems of Education (RISE) Working Paper.
World Bank, “Mobile cellular subscriptions (per 100 people)” (accessed 26 September 2018).
World Bank Global Report (2016), Engaging Citizens to Improve Service Delivery: The Citizen Feedback Monitoring Programme in Pakistan.