Generative artificial-intelligence tools are increasingly widely available.Credit: Nicolas Economou/NurPhoto via Getty
Around one in seven biomedical-research abstracts published last year was probably written with the help of artificial intelligence (AI), according to a massive analysis1 of the scholarly literature. More than 200,000 abstracts out of 1.5 million indexed in PubMed in 2024 contained words commonly suggested by large language models (LLMs).
The study was posted online as a preprint2 in June 2024, when it estimated that one in nine abstracts were written with AI assistance in the first half of last year. The updated analysis is published in Science Advances today1.
“The total number of LLM-edited papers has continued to rise inexorably,” says Andrew Gray, a bibliometrics support officer at University College London. He thinks researchers haven’t fully grasped the scale on which these tools are being used to produce scholarly work. “Hopefully, this paper will help drive some attention to the problem,” he adds.
Excess words
Many groups are trying to assess the impact of LLMs on scholarly output, but it is tricky because most users don’t disclose these practices. Efforts typically train models to identify differences between human-generated and LLM-generated text, and then apply those insights to assess the literature. But it is not clear how such models distinguish between the two types of text, and training data sets don’t always represent up-to-date trends in LLM-generated writing.
Dmitry Kobak, a data scientist at the University of Tübingen in Germany, and his colleagues took a more open-ended approach. They searched abstracts for ‘excess words’ that started to appear more often than expected after November 2022, when ChatGPT became widely available. The team was inspired by work estimating ‘excess deaths’ during the COVID-19 pandemic.
The researchers found that 454 words appeared much more often in 2024 than in any other year since 2010. They were mostly ‘style’ words unrelated to the content of the research, and tended to be verbs and adjectives. Some were common — such as ‘findings’, ‘crucial’ and ‘potential’ — whereas others were more unusual, including ‘delves’ and ‘showcasing’. Excess words that emerged in the second half of 2024 include ‘heighten’ and ‘hinder’, as well as superlatives such as ‘unparalleled’ and ‘invaluable’, says Kobak.
Shifts in the scientific lexicon happen over time — including dramatic changes that accompany major events, such as the COVID-19 pandemic that began in 2020. There were 190 excess words in 2021, and they tended to be nouns related to the content of the research, such as ‘mask’. But the lexical shift that has occurred since LLMs became popular has been even more pronounced, and mainly stylistic.
