How are we using AI at VoxDev?

I have been using generative AI extensively for over a year now, to help across a wide range of tasks at VoxDev. I have recently been asked about how I use these tools, so I thought I would outline exactly how we do use generative AI at VoxDev, where we can’t yet rely on it, and some of the debates we are having internally.

People are often surprised to find out that VoxDev only has two full-time staff, myself and Emaan Siddique, our excellent Deputy Managing Editor. I would like to think that our output is impressive for the two of us, as we oversee five to eight articles and one to two podcasts a week, plus a living literature review each month, while also hosting webinars and writing blogs. I think it’s fair to say that this level of output would not be possible without AI. Other caveats also apply; we are the only two full-time staff, but are supported by our colleagues at CEPR, Tim Philips and the Talk Normal Productions team for the podcast, our Editorial Board, and of course we are completely dependent on the brilliant community of development economics researchers who write about their research.

To summarise my reflections below, AI saves us a lot of time, but there is little we have completely handed over to AI - we still check any AI outputs we plan on using carefully before they make it to the website. As AI improves, I imagine this will slowly change, and our use-cases will expand. I end by getting existential about the role of a website like VoxDev in a world with widely used, highly powered generative AI models.

I use ‘AI’ and ‘Generative AI’ interchangeably below, which typically means ChatGPT as that’s our preferred LLM at this point. And no, this blog is not written by AI…

AI and editing

AI does a lot of boring tasks well, which saves us an enormous amount of time and allows us to focus on more interesting work. As a website with a house style, I spent far too long when I first joined VoxDev formatting references – I haven’t had to do this for over a year. AI also helps us by checking all in-text citations are in the reference list, and vice versa. It can also split reference lists by chapter depending on where work has been cited in the text.

AI is also very useful for minor copyedits. You might have noticed that we use UK spelling – AI can easily change the US spelling the majority of our authors use to UK spelling in seconds, with fewer mistakes than a human.

When asking generative AI to help with these tasks, we still have to put explicit guardrails in our prompts so it does not make changes we have not asked for. We are certainly not yet at the point where we can turn over the whole editing process. We have found that AI tends to sensationalise results, and can make mistakes when trying to summarise specific results. While you can continually prompt AI until these issues are fixed, it is much quicker for us to only hand over the simple tasks, while making more substantial edits on structure, tone and flow ourselves.

AI and writing

AI is great at creating well-written, nice-sounding summaries. However, an editor who understands the underlying research and can judge when generative AI might have embellished (it normally aggrandises) findings, or focused on the less interesting aspects, is still required.

I have also found that AI is very bad at writing introductions and conclusions. Maybe I haven’t figured out the correct prompts, but I do get the sense that these parts of the writing process which frame the article and takeaways still require a human touch.

So, how do I integrate AI into the writing process for my own blogs? Say I want to write about a specific theme based on a set of articles we have featured on VoxDev. Given that I’ve been here almost three years, I have a pretty good recollection of the articles we have on various topics, and can easily search the website to refresh my memory. But I still miss things, so often ask Deep Research (a feature on ChatGPT) to have a scan through the website and list everything it finds on a certain topic, as well as other similar sources elsewhere online.

Once I have some of the sources I want to highlight, since we do not want to get into the nitty gritty of econometric methods and rather focus on key takeaways for policymakers, generative AI does a decent job at summarising the research and how it relates to a topic. However, they are almost never ready to be entirely plugged in, so require editing – which I am well placed to do given the amount of development economics research I read.

When we approach someone to write an article for VoxDev, or accept a submission, we have no issue with them using generative AI in their writing process. Every article goes through the same review process and is read by myself and Emaan, so we are confident that we can edit these properly and catch any hallucinations (just like we catch the odd human mistake).

Surprisingly, I have not noticed a particular uptick in obviously AI-generated first drafts – I suspect this is because authors are using it effectively and properly editing afterwards, rather than not at all.

AI and podcast transcripts

As some of you have noticed, we have started producing write-ups for the podcast. We use AI in the following way:

• Otter AI produces a transcript of the episode.
• Copy and paste the transcript into ChatGPT, ask for a 1,200-word VoxDev-style summary including exact quotes from the transcript.
• Edit the first draft of text until we are happy it represents the research accurately, while double-checking the quotes are exact by relistening to the episode.

Adding these longer descriptions provides a useful summary of the podcasts, helps our website’s performance on Google, and now takes very little time.

AI and images

Improvements in AI-generated images mean we no longer have to rely solely on stock images – this is a big benefit for a platform like ours. AI image generation is not necessarily quicker than finding stock images, but it is often much better, particularly given that AI-generated images are increasingly easy to adjust. This is really useful as many stock images are stereotyped, exploitative and offensive.

I don’t anticipate we will move to completely AI-generated images, but will certainly increase the number we are using moving forward.

One useful trick for creating images is summarising some of the context you would like an image to capture, and asking ChatGPT to create three prompts that you could use. You can then select the one that sounds most appropriate and feed this back into the model - they are not always perfect but can be a good starting point if you are stuck for ideas on an appropriate image.

For example, for Stefan Dercon’s recent article “Best buys meet political realities: The political economy of education research”, ChatGPT suggested I use the following prompt:

• Create Image "An illustration depicting a path scattered with broken laptops leading toward a distant Kenyan school, capturing the symbolism of ambitious but abandoned technology projects. The landscape should reflect rural Kenya’s natural environment."

I thought this turned out well:

AIdeation

AI is very good at quickly churning out any number of ideas – they are not all good, and they almost never perfect, but this function is still very useful.

Say, for example, I want to give an article or podcast a title. Titles need to serve a number of purposes for a website like VoxDev, including accurately conveying the topic of research that’s being covered, encouraging a potential reader to click through, and fitting what a user might search on Google. This unnatural mix of objectives makes ChatGPT a useful assistant that can churn out titles. In some cases, one of the ten titles I ask it to create might work well and achieve all of our objectives. Most of the time, it points me in the direction of some combination of the phrases and words I should use, making the process quicker, and the title better, than I could have come up with alone with limited time.

Potential AI use cases and what we don’t allow AI to do

• Translation: This is a big one, and we are actively exploring how we can implement this effectively on the website.
• An AI search function for VoxDev content.

We are also very keen to hear your ideas/own experiences using AI. Let me know if you have any suggestions or reflections at [email protected].

Some important points, we do not:

• Let AI train on our inputs (although we do allow AI crawlers to access the VoxDev site).
• Use AI in our editorial review process.

The role of a website like VoxDev in the AI age

AI is dramatically changing how people search for information – this has big implications for those of us trying to communicate research to a wider audience.

Daily Traffic to VoxDev from Generative AI

In the past year, out of our 423,350 total users, about 1.6% (~ 6587) have come via generative AI tools, mainly ChatGPT, with Perplexity second and Gemini third. This is a small, albeit growing, proportion of our users which is currently outweighed by many other channels.

Generative AI presents a number of challenges for tracking our readership, and expanding our reach:

• With AI summaries at the top of Google searches, and other generative AI tools replacing Google search altogether for some users, the traditional ‘search query to website click through’ pipeline is becoming less dominant.
• Google has been, and still is, the most important channel for users finding VoxDev, but it is not hard to imagine Google search’s relatively outsized importance will change.
• There is no way of us tracking when generative AI users have been provided with answers to their questions using VoxDev content – and no good way of estimating this as it is hard to know what proportion click through to VoxDev from these platforms.
• Aside from some simple technical steps, it is difficult to know what determines which sources exactly AI prefers, and how we can ensure VoxDev is seen as a reliable source in their eyes, like we are by Google.

Beyond feeding AI models: How can websites stay relevant in an AI age

This is still a big unknown, so I can't say with any degree of certainty. But I have thought about this a lot. Here are a number of ways that a website like VoxDev might continue to be relevant in its own right, and not just a feeder of generative AI (with caveats and unanswered questions at the end).

• Our reputation: We have built up a reputation for high quality, accessible summaries of research over seven years, which certainly gives us an advantage.
• Academic rigour and policy relevance: Our Editorial Board reviews all underlying research for its quality and policy relevance – I suspect we still have a while before AI can match their expertise, especially at the margin.
• Experts: Leading experts will continue to be seen as valuable sources of information.
• What isn’t written down: Some of the most policy-relevant context and titbits from research featured on VoxDev might not have made the final version of the underlying academic paper.
• Using AI properly: We are careful when using AI so that hallucinations do not make it onto the website; this will be important in standing out amongst the AI slop and keeping trust.

Of course, whether these factors continue to matter depends on the pace of AI progress and diffusion. Part of the problem of planning for this future is that everyone has a different timeline - to the extent that it is already here, we are trying to work as quickly as possible.

Another key factor is people’s preference for advice/evidence from human sources rather than AI – will policymakers feel comfortable basing decisions on AI-generated answers and suggestions? We don’t have good current answers to this, or know how the answer might change over time as AI models evolve.