Education Technology

Assessments

Schools routinely need to test students for diagnosing learning levels, for tailoring instruction and feedback, and for certifying skill levels. Such assessments have been an important use-case for EdTech globally, but their potential in LMICs is perhaps even larger than in richer countries.

This is for two distinct reasons. First, as is widely documented, the actual achievement level of students is often severely misaligned with the grades they are enrolled in, and there is substantial variation in student achievement levels within individual grades (Muralidharan et al. 2019, Beg et al. 2024). In these settings, the ability for digital tests to adapt the items being administered to the ability of individual students — as is common in international assessments like the SAT and GRE — is a major advantage.

Second, existing evidence suggests that school-based assessments in LMICs can often be quite subject to substantial misreporting due to cheating by students or grade manipulation by teachers: for example, Singh and Berg (2024) document a doubling of reported achievement in administrative data in India versus responses to the same test questions by the same students in independent tests. Similarly, Johnson and Parrado (2021) show that official test scores in India’s National Achievement Surveys are unrealistically high and provide little reliable information about state rankings.[1]1. Test integrity is, of course, a concern in many settings beyond India. See, e.g. Berkhout et al. (2024) in Indonesia, Martinelli et al. (2018) in Mexico and Battistin et al. (2017) in southern Italy.

Digital testing, on tablets or computers, can potentially improve data integrity through at least three channels. First, different students can be presented with different items, which mechanically makes it harder for students to copy responses from each other. Second, since grading can often be automated, this reduces the possibility of teacher-induced grade manipulation. Finally, the availability of detailed item level data makes it possible to detect manipulation through an analysis of suspicious patterns.

Two recent studies provide empirical evidence highlighting the potential for digital testing to improve data integrity in student tests in LMICs. First, Singh (2024) presents results from a large experiment covering >2400 schools where roughly two-thirds were randomised to be tested on tablets, and one-third on paper; in a random subsample of 120 schools, students were re-administered the same test items in a common pen-and-paper format within two weeks of the initial tests to provide a common benchmark. He documents three results: (i) there is substantial cheating/manipulation in business-as-usual tests in this setting; (ii) this distortion is substantially reduced in tablet-based testing and (iii) direct audit-based measures of distortion correspond closely with the algorithmic procedure followed by Angrist et al. (2017) to detect cheating, which indicates that item-level data alone may be sufficient to detect class-level cheating, even where independent audits are not possible or too expensive. The study thus demonstrates both the usefulness and the feasibility of such assessments in LMICs within existing budgets (since tests can be staggered across classes and schools, enabling these to be carried out with a relatively small number of tablets).

Second, Berkhout et al. (2024) evaluate the impact of computer-based testing (CBT), an at-scale policy implemented by the Indonesian government to reduce widespread cheating in the national examinations. Exploiting the phased roll-out of the programme from 2015 to 2019, they report a sharp decline in test scores by 0.5 standard deviations after the introduction of CBT. Schools with response patterns that indicated cheating prior to CBT adoption experienced a steeper decline, and results do not differ by whether the school had a pre-existing computer lab (strongly suggesting that the decline is due to reduced cheating). Digital testing is already being scaled up in many high-stakes settings also in LMICs; the results in Berkhout et al. (2024) suggest that this could be substantially beneficial for improving the informativeness of these assessments.

Digital tests may also have other advantages. They can allow for tests that broaden the range of abilities being tested (e.g. by incorporating audio or video stimuli), can help improve data availability in developing countries, and also reduce the burden of grading from over-worked teachers. This is an active area of further research.

Incentives and other governance mechanisms

EdTech potentially also provides the pre-requisites for many interventions that target school governance. In particular, it can help make some previously-hard-to-measure inputs observable (e.g. effort by teachers or students) and can also make the measurement of outcomes more reliable.

An early example of such a use of technology in education systems was provided by Duflo et al. (2012), who used cameras to provide a verifiable measure of teacher attendance in schools, which they incentivise using a contingent-pay contract (i.e. teachers were paid more for higher attendance). Teacher absenteeism in the treatment group fell by 21 percentage points relative to the control group, and the children’s test scores increased by 0.17 standard deviations. While the intervention itself is about non-linear incentives tied to attendance, it is the technology (here, cameras) that allowed for a sufficiently robust basis to tie incentives to.

Similar issues arise with test scores, where the problems of widespread misreporting have been discussed above. Yet, measures of student achievement underlie many of the interventions that have appeared most promising in LMIC education systems. For example, performance-based pay in education has shown positive effects in large-scale RCTs in India, East Africa, and China (see, e.g. Muralidharan and Sundararaman 2011, Loyalka et al. 2019, Mbiti et al. 2019, Leaver et al. 2021, Gilligan et al. 2022, Mbiti et al. 2023), as have report-card interventions which distribute information about the performance of schools to the entire village (Andrabi et al. 2017, Afridi et al. 2020). In each of these experiments, however, the test data triggering payments were collected either directly by research teams or their NGO partners. One possible reason why these interventions have not scaled up more broadly is the unreliability of official assessments, which EdTech might help solve.

A further example is presented by the data on usage that is routinely provided by various EdTech products. In many interventions, the primary difficulty at scale is to sustain usage (and underlying dose-response relationships are stable). Since this is measured granularly and reliably, these could be the basis for direct student and teacher incentives on effort. See, especially, Hirshleifer (2017) for promising pilot evidence from an RCT in India.

Finally, there is substantial opportunity for the better use of available education data for decision-making, e.g. in resource allocation, personnel decisions and overall management. There is robust evidence that this correlates with school quality and school value-added (Bloom et al. 2015, Lemos et al. 2024), although interventions to improve school management using programmes that often feature an ICT component have been less successful (Muralidharan and Singh 2020). Similarly, many personnel decisions, e.g. on placement policies for teachers that build on a rich mechanism design literature (Combe et al. 2022, Bobba et al. 2021), rely on robust technology back-end. We do not review the many potential uses of technology in school systems on these dimensions.[2]2. These reforms could, of course, affect student outcomes downstream. Our choice to skip them in this review merely reflects a focus on technological applications where a focus on student achievement is more direct and the relevant technological tools are more education-specific rather than technologies with broad applicability to organisations across firms and the public sector.

For full reference list see the end of the conclusion chapter.