New IPN study demonstrates how AI helps generate systematic descriptions of research literature

The sum total of human knowledge is growing all the time – and so is the volume of academic and scientific papers and articles. Researchers survey an area of academic study by producing systematic summaries of the relevant literature. Yet this is an arduous and very time-consuming task.

A research team from the Leibniz Institute for Science and Mathematics Education (IPN) in Kiel, working with researchers from the universities of Hildesheim and Vienna, has now set out some ideas for speeding up this process. The researchers’ study, with the IPN’s Thorben Jansen and Lucas Liebenow as leads, has recently been published in the prestigious Psychological Bulletin and included in the Choice compilation issued by the American Psychological Association (APA) – a great honor for the team, confirming the quality of their work.

“The most onerous part of putting together a literature review is describing the properties of each individual article or paper,” says Lucas Liebenow. “This means two researchers compiling a table incorporating 50 to 100 pieces of information on the context, methodology, and findings of the study – for each and every paper.” Doing this manually can take up to three hours per paper or article. The research team looked at over 300,000 items of data relating to a total of 2,179 studies, with the aim of ascertaining the degree of precision with which eight AI models (of the type used in ChatGPT) carried out this task. They assessed precision by taking published literature reviews conducted by humans and having the AI models repeat the process.

The study found that the AI models’ precision varied from excellent to inadequate, with higher precision particularly likely to result where there was clarity on what information is needed and that information was set out unambiguously in the article or paper. Results were accurate more often when multiple models coded a piece of information than when only one model found it. This finding is indicative of the utility of using more than one model to carry out AI literature reviews.

Thorben Jansen comments: “Our study fundamentally demonstrates the potential of AI to help speed up enormously the process of putting together literature reviews. It used to take me a day to code two papers. Using AI, I can currently get through 15 to 20, and that’s definitely not the limit.” The team will now attempt to find out how researchers can use AI to make work with scientific methods faster while retaining full control of the process.

Read the open-access full text of the article at https://psycnet.apa.org/fulltext/2027-04635-004.html