The scientific community is adopting artificial-intelligence tools, especially large language models (LLMs), at an astonishing speed. LLM-assisted paper writing has drastically increased over the past three years1 and researchers have sought to incorporate semi-autonomous agents into their workflows. However, the rapid and uncritical adoption of AI in science comes with significant risks2. Several problems are already apparent: papers that use AI tools focus on a narrower set of established research questions3, and in some cases have been evaluated to have less scientific merit4, than do studies that do not rely on AI.
Why an overreliance on AI-driven modelling is bad for science
Moreover, as AI automates routine scientific tasks, concerns about the erosion of training opportunities for early‑career researchers remain largely unresolved and often unacknowledged. Conventionally, scientific training has combined formal instruction in facts and methods with the gradual acquisition of tacit knowledge through hands‑on, entry‑level work. Scholars in science and technology studies have repeatedly demonstrated that scientific texts alone do not fully communicate knowledge; instead, crucial know‑how is embedded in research communities and transmitted through apprenticeship and practice5.
This tacit knowledge — for example, of what constitutes ‘reasonable’ data, or the details of a technique that are difficult to articulate in a methods section, or whether a result is consistent with the existing literature — is essential if a researcher is to supervise AI‑assisted workflows effectively in the future. If AI systems increasingly replace entry‑level scientific labour, trainees might never develop these skills, potentially leaving the next generation of scientists ill prepared to oversee AI‑driven research responsibly.
These trends demand that scientists reckon with the purpose of scientific institutions. Is our goal simply to build a collection of scientific facts, or to also cultivate a living, evolving community of scientific knowers? If AI tools accelerate the former goal while threatening the latter, how should scientists proceed? Drawing on emerging evidence about AI’s effects on scientific practice, here we highlight key risks and outline potential remedies.
More output, less understanding
The AI industry has aggressively marketed LLM products to scientists as technologies for increasing productivity. Some researchers have embraced this promise, touting these products’ ability to ‘supercharge’ their academic writing6.
Is AI leading to a reproducibility crisis in science?
If we take productivity to mean the output of scientific papers, AI products have undoubtedly delivered on the promise1,7, with far-reaching effects. Online preprint repositories such as SocArXiv and PsyArXiv imposed temporary moratoriums on AI papers or updated their reviewing policies; arXiv changed its policy and no longer accepts computer-science position papers (which are relatively easy for LLMs to churn out), and the US National Institutes of Health began limiting grant proposals to six per year per principal investigator (PI).
Alongside increased productivity, there is evidence that scientists who use AI tools in their work receive more citations and advance more quickly to PI roles than do those who do not3, with people who have English as their first language benefiting disproportionately8. It’s noteworthy, however, that the career benefits of AI products are most apparent in quantitative metrics, such as publication and citation counts. Such measures cannot account for whether these benefits accrue because of scientific contribution or as a result of connections with a hyped field. It would therefore be a mistake to take these metrics as representing a win for science, without looking more closely at the quality of this increased productivity.
A robot learns to emulate human behaviour at a facility in Hefei, China.Credit: Zhang Dagang/VCG via Getty
More is not necessarily better, and there is growing evidence that LLM-driven increases in scientific productivity often come at a cost. Most obvious is the introduction of ‘AI slop’ into the literature, including nonsensical AI-generated images and hallucinated citations.
Multiple studies have documented troubling trends in papers that heavily rely on LLM assistance in writing. The journal Organization Science audited all of the 6,957 submissions it received from January 2021 to January 2026, and observed that LLM-assisted papers had poorer scientific quality (as measured by the likelihood of these works being accepted by the journal)7. Another study4, of 264,125 papers gathered from a 2024 AI conference and submissions posted across three preprint servers (arXiv, bioRxiv and SSRN) in 2023–24, found that in LLM-assisted papers, good writing ceased to be an accurate heuristic for scientific quality (as measured by publication outcome and peer-review scores).
Can AI have common sense? Finding out will be key to achieving machine intelligence
The authors conclude that this creates “a risk for the scientific enterprise, as a deluge of superficially convincing but scientifically underwhelming research could saturate the literature”. Currently, there are no clear solutions to these pollutions of our knowledge ecosystem, despite agreement that they should not be happening.
AI tools implemented across the research pipeline are also limiting scientific understanding in more subtle ways, for example, by shrinking the set of concerns that researchers explore. An analysis3 of 41.3 million research papers spanning biology, medicine, chemistry, physics, materials science and geology shows that the adoption of AI seems to “induce authors to converge on the same solutions to known problems rather than create new ones”. This is an early warning of the likelihood that AI will have cascading impacts on the knowledge ecosystem2.
Threat of deskilling
The loftiest visions of AI for science involve semi- or fully autonomous ‘AI scientists’9, and recent prototypes of this vision have received great fanfare in science and industry alike. Developers of AI scientists are quick to reassure that their intention is “not to replace human scientists, but to augment and accelerate their work”10. These products, then, are meant to be overseen by a skilled scientist who can responsibly direct and check the AI’s output.
We question whether this vision can come to pass without a clear articulation of how the responsible scientist can develop the expertise to distinguish between good and poor AI outcomes. Much of the enthusiasm for AI tools across the scientific pipeline comes from their promise to offload work. But many ‘low-skilled’ tasks have conventionally been important starting points for trainee scientists.
