Credit: WIN-Initiative/Neleman/Getty
Cancer research articles with telltale signs of being produced by paper mills garner double the number of citations than do genuine papers in the field, finds an analysis of tens of thousands of articles1.
In a study posted on the preprint server bioRxiv, the authors report that papers that were probably produced by paper mills frequently cite, or are cited by, other potentially fraudulent articles. Paper mills are businesses that produce and sell low-quality manuscripts — often containing fabricated data and results — designed to resemble genuine research.
Adrian Barnett, a statistician at the Queensland University of Technology in Brisbane, Australia, and his colleagues say that their analysis indicates that coordinated citation manipulation is inflating the impact metrics of journals in molecular oncology.
These metrics measure of how often a journal’s papers are cited in other research, among other things. In many nations, having papers published in journals with high impact factors is taken into account when researchers apply for jobs and funding.
Research-integrity sleuths have long suspected that paper mills are inflating citations, says René Aquarius, a neurosurgery researcher at Radboud University Medical Center in Nijmegen, the Netherlands. “But it’s nice to see this confirmed in such an elegant way,” he adds.
Leading journals not immune
Barnett says that there is an assumption that studies produced by paper mills are only being published in journals with low impact metrics, but his team’s research suggest that this is not the case.
Barnett and his colleagues analysed 33,159 papers published between 2012 and 2023 in 20 high-impact molecular-oncology journals. The team used an artificial-intelligence tool called BERT that they had developed previously to identify suspicious papers. It looks for telltale characteristics that often appear in retracted paper-mill articles. Paper mills often mass-produce manuscripts that use fabricated data sets, manipulated images and unusual or ‘tortured’ phrases that are designed to evade plagiarism detectors. For each paper it reviewed, BERT assigned a probability score estimating how likely the study was to have been produced by a paper mill.
The tool flagged 4,085 papers — 12.3% of all of the papers examined — as having characteristics associated with paper-mill articles. The researchers identified potentially fraudulent papers in 19 out of the 20 journals that they examined. Nature Cancer was the only journal in the analysis that had not published any suspicious papers. (Nature’s news team is editorially independent of its publisher, Springer Nature, which also publishes Nature Cancer.)
Adam Day, the founder of research-integrity firm Clear Skies in London, which helps publishers to find out whether their journals are being targeted by paper mills, says that detecting patterns of inflated citation counts helps spot journals at risk.
Researchers have noted problems with BERT in a previous study, such as detecting false positives2 — papers that were genuine. It’s unclear whether BERT’s false-positive rate affects the latest study results, says Day.
Barnett says that the 4,085 flagged papers probably include some genuine studies. But given the large sample size, these will not change the general patterns seen in their findings. He notes that the tool is meant to point towards probable suspects and general patterns of paper-mill activity — rather than definitively determine whether a paper is genuine or not.
Gaming citations
The team also examined the citation patterns of suspicious papers and the journals in which they were published. They found that the suspect articles had been cited by other papers up to twice as many times as papers that had not been flagged. The difference between the citation rate of suspicious and non-suspicious papers was highest in the first few years after publication, but dropped off over time.
The researchers also identified a pattern of suspicious papers citing other suspicious papers. The authors suggest that paper mills are citing their previous papers. “When a dishonest researcher buys a paper mill product, they not only get a published paper, but also get citations to that paper as part of the bargain,” says Barnett.
In some journals, suspicious papers accounted for more than half of the citations that they received. For example, for Molecular Cancer and the Journal of Experimental & Clinical Cancer Research, 57% of citations came from flagged papers. (Both journals are published by the BMC portfolio, which is owned by Springer Nature.)
Selene Carey, the publishing director for BMC journals, said that she and her team are reviewing the preprint’s findings and will take action where appropriate.
