The value of reference letters: Experimental evidence from South Africa

Information asymmetries about workers’ skills are prevalent in labour markets, especially in the market for low-skill and entry-level jobs. This can have adverse effects on match quality, ultimately affecting output and employment. To the extent that employers are more uncertain or underestimate the ability of disadvantaged groups, such as women, information frictions can also exacerbate labour market inequality (Agrawal et al. 2016, Lang and Manove 2011). Hiring firms can partially reduce these asymmetries through referrals from previous employers, who may have valuable information about workers’ skills that is otherwise unobservable to them. However, in various contexts (particularly in the developing world) this practice is largely absent.

Reference letters 

In Abel et al (2019), we test the effect of reference letters on a sample of young South African job seekers in low-skill sectors. At the baseline, fewer than 5% of our sample use a reference letter. The existing letters typically lack essential information like the assessment of specific skills or the referee’s contact details. Our first task was therefore to design a reference letter template. To do so, we asked employers two questions: What are you interested in seeing in a letter? And what information are you willing to provide? This resulted in a one-page template (as shown in Figure 1) that asks employers to rate a range of hard and soft skills and to provide information on the employment relationship.

Figure 1 Reference letter template

Does the letter help job seekers and firms?

In order to test whether the letter is effective and, if so, why, we conducted a series of field experiments in collaboration with the South African Department of Labour (DoL).

In our first experiment, we encourage a group of low-skilled job seekers with some prior work experience to ask a former employer to complete the template. About one-third of them returned the completed letter to us. 

For this sample, we conduct an audit study in which we submit applications on behalf of the job seeker and randomly add the reference letter to the application material. The within-person analysis reveals that the reference letter increases response rates by 60%. This effect is driven by female participants whose call-back rates approximately double.

This result is corroborated in a second experiment in which we encourage a random half of a separate group of job seekers to obtain a letter and use it in their job search. Three months post-intervention, employment rates for women increase by about 50%, while we find no effects for men. Showing that reference letters increase employment after people adjusted their search strategy is of particular policy relevance as the encouragement design is easily scalable.

Importantly, in our first experiment we also show that reference letters help firms in the selection of candidates. Job seekers selected from applications with a letter attached perform significantly higher on an aptitude test administered by us (but not visible to the employer). Although our design cannot explicitly test for general equilibrium effects, theory predicts that the identification of higher ability workers should increase firm demand (Wolpin 1977). 

Why are reference letters effective?

Does the letter content matter to the firm or is it merely signalling that the applicant is the type of person who can obtain one? We shed light on this question by testing whether more positive letters are indeed more effective. The red line in Figure 2 shows the call-back probability as a function of how positive the letter is. Importantly, through the within-person randomisation, applications without the letter provide us with a counterfactual (blue line). The graph shows that the effectiveness of letters increases as they become more positive. However, letters with a top rating in every skill category are ineffective. We interpret this as a perceived lack of credibility.

Figure 2 Measuring the importance of a positive letter

Are firms right to pay attention to the letter content? We find that the employer rating of workers’ literacy and numeracy skills are highly correlated with scores from an aptitude test (unobservable to the employer) in the same domain. Consistent with the finding on improved applicant screening, this suggests that letters provide hiring firms with reliable information about the candidates’ abilities. Arguably, the content is even more important for soft skills, which are increasingly important, but harder to assess at the screening stage (Demming 2017).

This points to a potential key advantage of using former employers for referrals over co-workers. While current workers also hold valuable information on the productivity of their peers (Pallais and Sands 2016), they may be reluctant to pass on truthful information unless sufficiently incentivised (Beaman and Magruder 2012).

Why do women benefit more from reference letters?

Reference letters for female job seekers are not more positive, nor is the letter content more highly correlated with ability for women. Our empirical findings point to two potential explanations for why letters may be more effective for women.

• First, we show that women are significantly more likely to use letters in their job search. This is consistent with the idea that groups who feel in a position of disadvantage are more likely to embrace additional tools to prove their ability.
• Second, the letters’ content is more highly correlated with the probability of receiving an employer response for female job seekers. This suggests that firms are more uncertain about the ability of female applicants.

Policy implications

Why are reference letters largely absent despite positive gains for job seekers and firms?  One simple explanation is that writing an accurate reference letter is costly and previous employers do not benefit from it. We may have addressed this barrier by providing an easy-to-fill template.  

A second explanation is that job seekers themselves are not motivated to obtain letters. In a third experiment, we indeed show that providing information on the benefits of reference letters significantly increases the share of participants that obtain one, suggesting that job seekers underestimated their effectiveness. 

Overall, the study shows that information asymmetries are prevalent in low-skill labour markets and that employers struggle to identify high-ability job seekers. We show that a simple intervention leveraging information from former employers can reduce these asymmetries and improve firms’ screening ability. In addition, we show that equipping women with a tool to signal their skills may improve their employment prospects. Reducing information frictions can thus contribute to levelling the playing field for job applicants at an initial disadvantage. Taken together, these results may provide a rationale for governments to facilitate the information exchange.

The South African DoL is currently in the process of scaling up this intervention as part of their counselling programme, providing job seekers with both the template and information on its usefulness.

Editor’s note: This column is part of our series on information in the labour market.

References

Abel, M, R Burger and P Piraino (2019), “The Value of Reference Letters: Experimental Evidence from South Africa”.

Agrawal, A, N Lacetera and E Lyons (2016), “Does standardised information in online markets disproportionately benefit job applicants from less developed countries?”, Journal of International Economics 103: 1–12.

Beaman, L and J Magruder (2012), “Who gets the job referral? Evidence from a social networks experiment”, The American Economic Review 102(7): 3574–3593.

Deming, D (2017), "The growing importance of social skills in the labour market." The Quarterly Journal of Economics 132(4): 1593-1640.

Lang, K and M Manove (2011), “Education and labour-market discrimination”, American Economic Review 101(4): 1467–1496.

Pallais, A and E Glassberg Sands (2016), "Why the referential treatment? Evidence from field experiments on referrals", Journal of Political Economy 124(6): 1793-1828.

Wolpin, K (1977), “Education and screening”, The American Economic Review 67(5): 949–958.