Improving learning efficacy and equality with AI training

A crucial task for researchers is identifying how educators and policymakers can improve educational outcomes and create an inclusive and equitable learning environment. Evidence shows that a number of practices can hinder students from reaching their full potential, particularly discriminatory practices and stereotypes identified in classrooms which lead to gender performance gaps later on (Alan et al. 2018, Lavy and Sand 2018, Carlana 2019). Gender discrimination can emerge as early as primary education (Ganley et al. 2018, Breda et al. 2020), and promoting gender equality in education has been a critical social goal for an extended period (Klasen 2002, Duflo 2012, Gallus and Heikensten 2020). However, teachers and mentors may harbour gender-biased beliefs, which can be slow to change. The advancement of AI technology offers a potential solution to this issue. AI has already been extensively utilised for educational purposes, and gender-related variables can be removed from AI input to enhance gender neutrality. Nonetheless, there is limited evidence on the effectiveness of AI training, or on its potential benefits in reducing gender performance gaps.

An experiment with AI Teachers

In our study, we analyse data from an experiment where AI replaced a random subset of human teachers at a training agency for Go, an abstract strategy board game for two players. This field setting is ideal for a comprehensive examination of AI teaching for several reasons. First, students must attend two training sessions per week, each consisting of a tournament and a match review with their teachers. This straightforward training structure allows us to study how a teacher’s guidance during reviews can enhance students’ Go skills in subsequent games. The frequent gameplay provides ample observations to assess students’ skill development and draw statistical inferences. Second, Go is a male-dominated game with a pre-existing gender performance gap, enabling us to explore the gendered impacts of AI training. Third, due to China’s COVID prevention regulations, both tournaments and reviews were conducted online during the analysis period. The Go-playing software recorded all moves, facilitating the quantification of student performance. We also recorded review sessions led by both human and AI teachers to investigate the relationship between teachers’ facial, vocal and verbal characteristics, and student learning outcomes.

It is important to note that the AI teacher is depicted as a cartoon character on the screen, providing facial, vocal, and verbal information for analysis, similar to human teachers. Furthermore, student-teacher interactions occur online for both human and AI teachers, ensuring a fair comparison between the two teaching methods. Additionally, the AI does not collect any student information (e.g. gender) beyond the moves made in each tournament.

AI teaching outcomes

We conducted a comprehensive analysis of tournament data for two groups: students taught solely by human teachers (control group) and students who received training from the AI teacher as a result of the intervention (treated group). Before the introduction of AI training, both groups exhibited parallel performance trends in Go. However, after the intervention, the treated group demonstrated significantly faster improvement in Go skills compared to the control group, as shown in Figure 1. This suggests that the AI teacher is more effective than human teachers in enhancing students’ learning outcomes across the entire student population.

Figure 1: Winning ratio

Notes: This figure illustrates the winning rates (raw rates) of the control and treated groups before and after the experimental intervention. The vertical axis represents the winning probability, while the horizontal axis displays the time relative to the commencement of the intervention. The dashed error bars indicate the 95% confidence interval.

Next, we examined the impact of AI teaching on boys and girls separately. As shown in Figure 2, we observed a persistent gender performance gap before the experimental intervention, with boys performing significantly better than girls. After introducing AI, we found that both genders in the treated group progressed faster than their counterparts in the control group, with girls advancing more rapidly than boys under AI training. Among the treated group, boys and girls achieved similar performance after five months of AI training.

Figure 2: Winning ratio by gender.

Notes: This figure demonstrates the winning rates (raw rates) of the control and treated groups before and after the experimental intervention, categorised by gender. The vertical axis represents the winning probability, while the horizontal axis displays the time relative to the commencement of the intervention. The dashed error bars indicate the 95% confidence interval.

How does AI teaching affect learning?

In the next step, we surveyed students who experienced both AI and human teaching on teaching-related questions to understand the potential channels driving the treatment effects. Survey results show that AI’s ability to analyse games, provide relevant statistics, and offer interactive features helped students learn faster than those taught by human teachers. However, we did not find any significant gender differences in responses to these questions.

We further analysed video recordings of all revision classes to study the reasons behind the initial gender performance gap and how AI training can reduce it. The data indicates that human teachers displayed more positive and fewer negative emotions towards boys and students with more advanced Go skills. The data also reveals significant correlations between teachers’ emotional status in the revision class and students’ performance in subsequent games. On the other hand, the AI teacher’s emotions exhibited less variation than human teachers and were not gender dependent. Consistent with the video evidence, the survey results indicate that girls picked up on the gender-biased emotions exhibited by human teachers, which may have influenced their learning outcomes and might explain the evolution of the gender performance gap. Students did not perceive the AI teacher as exhibiting gender bias.

Takeaways

AI teachers’ inherent advantages in analysing game data, providing relevant information, and friendly interactive features help students learn faster, and their gender-neutral emotions can help reduce the gender-performance gap. Similarly, the advantages of incorporating AI into education can extend to diverse domains such as Science, Technology, Engineering, and Mathematics (STEM), along with language arts. This is particularly relevant as STEM subjects are commonly associated with masculine stereotypes, while language-related classes are often perceived as more feminine.

Beyond the educational setting, we believe the results may apply to the training of managers and policymakers as decision-making in Go mirrors that of managers and policymakers, who rely on intuition to solve complex problems under uncertainty and time constraints (Miric et al. 2021, Mnih et al. 2015). Moreover, AI excels in detecting discrimination and subtle emotional changes, offering a means to mitigate bias and gather valuable information when participants’ emotions reveal their thoughts during business negotiations, court hearings, and political discussions. Consequently, our findings have implications for AI’s instructional role in enhancing decision-making for managers and policymakers.

References

Alan, S, S Ertac, and I Mumcu (2018), "Gender stereotypes in the classroom and effects on achievement", Review of Economics and Statistics, 100(5): 876–890.

Bao, Z, D Huang, C Lin (2024), "Can Artificial Intelligence Improve Gender Equality? Evidence from a Natural Experiment" Management Science, forthcoming.

Lavy, V and E Sand (2018), "On the origins of gender gaps in human capital: Short- and long-term consequences of teachers’ biases", Journal of Public Economics, 167: 263–279.

Carlana, M (2019), "Implicit stereotypes: Evidence from teachers gender bias", The Quarterly Journal of Economics, 134(3): 1163–1224.

Ganley, C M, C E George, J R Cimpian, and M B Makowski (2018), "Gender equity in college majors: Looking beyond the STEM/non-STEM dichotomy for answers regarding female participation", American Educational Research Journal, 55(3): 453–487.

Breda, T, E Jouini, C Napp, and G Thebault (2020), "Gender stereotypes can explain the gender-equality paradox", Proceedings of the National Academy of Sciences, 117(49): 31063–31069.

Klasen, S (2002), "Low schooling for girls, slower growth for all? Cross-country evidence on the effect of gender inequality in education on economic development", The World Bank Economic Review, 16(3): 345-373.

Duflo, E (2012), "Women empowerment and economic development", Journal of Economic Literature, 50(4): 1051–1079.

Gallus, J and E Heikensten (2020), "Awards and the gender gap in knowledge contributions in STEM", AEA Papers and Proceedings, 110: 241–44.

Miric, M, M Pagani, and O A E Sawy (2021), "When and who do platform companies acquire? Understanding the role of acquisitions in the growth of platform companies", MIS Quarterly, 45(4): 2159–2174.

Mnih, V, K Kavukcuoglu, D Silver, A A Rusu, J Veness, M G Bellemare, A Graves, M Riedmiller, A K Fidjeland, G Ostrovski, S Petersen, C Beattie, A Sadik, I Antonoglou, H King, D Kumaran, D Wierstra, S Legg, and D Hassabis (2015), "Human-level control through deep reinforcement learning", Nature, 518(7540): 529–533.