Researchers looked at claims from 400 papers about Drosophila immunity published between 1959 and 2011.Credit: Pascal Goetgheluck/Science Photo Library
A huge project has analysed more than 1,000 claims about the immunity of Drosophila fruit flies in scientific papers published over some 50 years, and has found that at least 61% are verifiable.
The work, described in two preprints posted to bioRxiv on 9 July1,2, shows that the field is in good shape, says co-author Mark Hanson, who studies the evolution of Drosophila immune systems at the University of Exeter, UK.
“There is a narrative of a ‘reproducibility crisis’ in science, which sounds dramatic,” he says, but adds that the scientists themselves are rarely that worried. “Fly genetics is so incredibly powerful, and allows so many independent experiment types, that we constantly build on each other’s work all the time.”
Testing claims
The idea of a reproducibility crisis stems from concerns that some scientific work is difficult to reproduce by other researchers. There have been many efforts to validate individual studies and key findings, some with shocking results.
For example in 2015, it was reported that more than half of 100 psychology studies fail reproducibility tests3, and earlier this year, a coalition of 56 research teams surveyed dozens of Brazilian biomedical studies and could replicate the results of less than half of the tested experiments.
Huge reproducibility project fails to validate dozens of biomedical studies
In the latest preprints, Hanson, with Bruno Lemaitre, a geneticist who studies fruit-fly immunity at the Swiss Federal Technology Institute of Lausanne (EPFL), and their colleagues tried to assess the reproducibility of their entire research field.
“This started with the observation that there was no follow-up of a number of big claims in big papers,” says Lemaitre, who led the study.
The team looked at 1,006 results from 400 papers published in the field between 1959 and 2011 — up until Jules Hoffmann won a share of the 2011 medicine Nobel prize for work on the Drosophila immune response.
They then went through the published literature looking for later work that either supported or refuted each claim. Rather than looking for studies that exactly replicated the methods of an experiment behind a result, or reproduced an identical effect, they searched to see if the claims were successfully built on by other studies, were broadly reproduced using more-modern techniques or still stood up to scrutiny now that knowledge of the field has moved on.
Using their expertise of the field, the researchers classified claims as either verified, unchallenged by other papers, challenged or partly verified.
They found that 61% of the 1,006 claims were verifiable, whereas only 7% had been challenged and weren’t reproducible.
Replication games: how to make reproducibility research more systematic
Of the 240 claims that had never been independently tested, the team could further categorize a subset of 111 as consistent with current knowledge and 22 as inconsistent with it. They performed experiments to test a further 45 claims, and found 38 of them to be inconsistent with current knowledge. “These were claims that I knew were a bit fragile,” says Lemaitre.
