Up to one in 50 scientific papers ought to be retracted, according to one estimate.Credit: Brazzo/Getty
The Center for Scientific Integrity — the non-profit organization that runs Retraction Watch — has launched a project aimed at rounding up flawed and fake medical-research papers and neutralizing their impact on health guidelines. The Medical Evidence Project will be run by James Heathers, a science-integrity consultant and self-described “data thug” based in Boston, Massachusetts.
Medicine is plagued by untrustworthy clinical trials. How many studies are faked or flawed?
“I don’t think anything is more of an immediate threat to people than bad medical research,” Heathers says. “We’re going to find the bad medical evidence that is a threat to quality of life and tell everyone.”
The project, which has a US$900,000 grant from funder Open Philanthropy in San Francisco, California, to run for two years with a team of three to five people, aims specifically to root out flawed papers that have a serious impact on medical guidelines by skewing meta-analyses — reviews that combine the results of multiple similar studies to come to a statistically more powerful conclusion. “It is a great idea to take a very detailed look at such papers,” says Elisabeth Bik, a data sleuth in San Francisco who is not involved with the project. Earlier this year, the Center for Scientific Integrity (CSI) set up a fund for grants under Bik’s name after she donated $200,000 to the organization.
Ivan Oransky, executive director of the CSI, says that the centre has “been expanding and embracing the sleuth movement” by helping to fund and support work that is usually done on a volunteer basis.
Flaws and fakery
Research papers marred by innocent mistakes, misconduct or outright fakery are a serious problem in science. Oransky estimates that one in every 50 papers published by scientific journals should probably be retracted. That amounts to tens of thousands of papers among the roughly three million published each year.
Giant study finds untrustworthy trials pollute gold-standard medical reviews
Other projects have already sought to root out bad data. One trawled through 50 high-profile medical reviews and found serious concerns with about 6% of the papers that fed into them1. The team that ran that project is working on a tool to help check papers for trustworthiness. The historical examples of papers such as these affecting health policy “are terrifying”, Heathers says.
For example, a 2009 guideline from the European Society of Cardiology2 recommended the use of blood-pressure-lowering medicines called beta-blockers to protect the heart during non-cardiac surgery, based heavily on a series of studies that began in 1999. However, a few years later, concerns were raised about those studies. The guideline was amended in 2013 with an ‘expression of concern’3, but a 2014 investigation was unable to “confirm or dispel doubts” about the work. Other researchers reported their own analysis4 of the literature that year and suggested that adherence to that guideline might have caused up to 10,000 deaths per year in the United Kingdom alone.
The Medical Evidence Project, which launched on 1 June, wants to provide a secure platform for whistle-blowers who are daunted by the idea of publishing their concerns about influential medical papers on public platforms such as PubPeer, or find that route ineffective. “I would like to be a resource for those people,” Heathers says. The project’s website will provide a mechanism to supply anonymous tips. People can also e-mail the project’s tip line.
The team plans to develop its own computational methods for finding bad papers — perhaps similar to the Problematic Paper Screener, a tool that looks for red flags such as papers that cite retracted works or that use ‘tortured phrases’, hinting that they have been written by automatic means. But the team will fine-tune its tools for their own purposes.
