Novelty is often espoused as a virtue in science. But what does it mean, and how valuable is it, really? A manuscript written in gobbledegook might be considered novel, despite being useless. And studies that try to replicate previous work can be valuable without being novel.
Some scientists argue that the current research system, driven by citation-based metrics, incentivizes researchers to play it safe. It’s easier to get citations for incremental work in established fields than to break out of silos. Others counter that the existing system demands novelty at the expense of depth and replication, and that what’s considered novel can be subjective.
‘Disruptive’ science has declined — and no one knows why
As the co-leader of the UK Metascience Unit, I’ve grappled with this debate. It’s important to consider whether more novelty would make research more impactful. But I’ve been stumped by the fact that there are no good ways to measure novelty. Without good indicators, researchers can’t assess the prevalence of original papers or their value in scientific progress.
Metascientists have tried to assign novelty scores to academic papers — and their authors — in various ways, none of which are foolproof. One identifies early users of certain key words, so we can see retrospectively who were pioneers of ideas. Another looks at how scientific papers cite each other; novel research often cites combinations of papers that had not been drawn together previously.
In the past few years, artificial intelligence (AI)-based models have emerged that analyse the textual similarity between a paper and the existing research corpus. By ingesting large amounts of text from online manuscripts, these models have the potential to be better than previous models at detecting how original a paper is, even in cases in which the study hasn’t cited the work it resembles. Because these models analyse the meanings of words and sentences, rather than word frequencies, they would not score a paper more highly simply for use of varied language — for instance, ‘dough’ instead of ‘money’.
Pay researchers to spot errors in published papers
But these AI-based novelty indicators, and, to an extent, their predecessors, are yet to be robustly validated. Do they cohere with human judgements of what novel science is? Would they single out work that went on to win Nobel prizes or to be funded by novelty-focused programmes such as those by DARPA and ARIA, say? Without testing them at scale across multiple fields, we cannot trust them.
That’s why the UK Metascience Unit has partnered with the non-profit organization RAND Europe; the Sussex Science Policy Research Unit; and the publisher Elsevier, to launch MetaNIC (see go.nature.com/3hhsdp3) — a competition to produce and validate indicators for scientific novelty in academic papers. Running until November, MetaNIC is open to researchers all around the world.
Participants will design novelty assessments and test them over a set of 50,000 research papers, drawn from many fields. We will compare the algorithms’ scores against a huge ‘ground truth’ data set, gathered by asking more than 10,000 researchers to assess the novelty of the same set of papers, each being assigned only studies in their own field to judge. The team behind the indicator that best aligns with the human judgements will be awarded £300,000 (US$407, 000) to further develop their work.
Once we have an indicator that reliably matches human judgements, metascientists will be able to study the role of originality in scientific progress.
