Credit: Denis Borisov/Getty
Researchers are increasingly using autonomous artificial-intelligence agents for data interpretation. The accuracy of this analysis depends on the data sets that are available to these programs, and a new study documents how easy it is for fraudsters to ‘poison’ these resources by uploading manipulated data sets that closely resemble the original ones but lead to different conclusions.
“Attacks of this nature are almost inevitable, because they enable people to launder false information through an authoritative-sounding filter,” says Vitaly Shmatikov, a computer scientist at Cornell Tech in New York City. The authors say that when using AI to analyse online data, researchers should be very careful to check the provenance of the resources they are mining.
In their analysis1, posted on the arXiv preprint server on 12 July, the authors downloaded public data sets concerning five controversial issues. They then tweaked the data sets to change the direction and strength of statistical trends, and uploaded the manipulated versions to private repositories while making sure that neither the public nor autonomous AI systems could access the incorrect data outside the study.
Finally, the team granted access to AI agents built by Anthropic, OpenAI and Google and asked them to search both public and private repositories and use the data sets to answer questions. Around half the time on average, the researchers found, the AI agents fell for the manipulated data sets and arrived at the conclusion that the ‘fraudsters’ wanted. (The article has not yet been peer reviewed.)
Controversial issues
The manipulated data sets covered current, highly charged issues: the relationship between immigration and fertility rates in the United Kingdom and the European Union; discrimination in workplace recruitment; racial disparity in policing; safety comparisons between human drivers and autonomous vehicles; and the impact of generative-AI use on worker motivation.
Because they touch on societal issues, these data sets represent potential targets for misinformation campaigns, says study co-author Nihar Shah, a computer scientist at Carnegie Mellon University in Pittsburgh, Pennsylvania.
But data poisoning is not unique to scientific data. Shmatikov co-authored a study, posted on arXiv in May, that found that it’s easy to use online forums such as Reddit to manipulate the output of AI agents2. And in April, Nature reported on a study in which AI chatbots started warning people about a made-up eye disease after researchers uploaded fake studies about the condition.
NatureTech
Brian Nosek, executive director of the Center for Open Science and a psychologist at the University of Virginia in Charlottesville, says the latest study highlights how AI systems can be manipulated to undermine scientific integrity.
“This is a demonstration of what could happen,” Nosek says, “as we rely on AI systems to do some of the thinking and doing for us.” When examining the performance of AI agents, “it can be easy to miss things that are leading us astray, and the lack of attention to [data] provenance is a critical one”, he adds.
Although the AI agents in the study pulled information from both the original data sets and the manipulated ones, it’s possible to make them disregard the originals entirely, Shah notes. Fraudsters can do this by tweaking the ‘README’ files, which describe and explain the data sets, to indicate that agents should ignore the original versions because they contain errors.
