Tuesday, September 9, 2025
No menu items!
HomeNatureAI chatbots are already biasing research — we must establish guidelines for...

AI chatbots are already biasing research — we must establish guidelines for their use now

Artificial intelligence (AI) systems are consuming vast amounts of online content yet pointing few users to the articles’ publishers. In early 2025, US-based company OpenAI collected around 250 pages of material for every visitor it directed to a publisher’s website. By mid-2025, that figure had soared to 1,500, according to Matthew Prince, chief executive of US-based Internet-security firm Cloudflare. And the extraction rate of US-based AI start-up company Anthropic climbed even higher over the same period: from 6,000 pages to 60,000. Even tech giant Google, long considered an asset to publishers because of the referral traffic it generated, tripled its ratio from 6 pages to 18 with the launch of its AI Overviews feature. The current information ecosystem is dominated by ‘answer engines’ — AI chatbots that synthesize and deliver information directly, with users trusting the answers now more than ever.

As a researcher in metascience and psychology, I see this transition as the most important change in knowledge discovery in a generation. Although these tools can answer questions faster and often more accurately than search engines can, this efficiency has a price. In addition to the decimation of web traffic to publishers, there is a more insidious cost. Not AI’s ‘hallucinations’ — fabrications that can be corrected — but the biases and vulnerabilities in the real information that these systems present to users.

Consider what happens when researchers ask an AI tool to recommend peer reviewers in their field. One study focused on physics found that AI systems over-represented scholars with names that the scientists classified as belonging to white people and under-represented those with names they classified as Asian (D. Barolo et al. Preprint at arXiv https://doi.org/p46k; 2025). Algorithms can amplify distinct prejudices depending on the field, the context and the wording of the query.

The distortion extends to the literature itself, something I’ve witnessed at first hand. When AI systems recommend research papers, they consistently exacerbate the Matthew effect — the tendency for highly cited work to accumulate even more citations while lesser-known research remains invisible. More than 60% of AI-generated suggestions fall in the top 1% of most-cited articles, more than twice the number seen in human-curated reference lists (A. Algaba et al. Preprint at arXiv https://doi.org/p46m; 2025). AI systems have internalized human citation patterns and have amplified them to an extreme.

Despite the tools’ growing influence, research on AI-assisted information retrieval remains limited. This gap reflects a broader blind spot in how institutions are responding to AI’s growing role in science. Policies have focused on the ethics of AI-assisted article production — establishing principles for responsible use, concerning originality, accountability and transparency, for instance. However, the threat lies not in how we write science, but in how we find it.

Researchers are scrutinizing AI-generated sentences while implicitly enabling these systems to choose which scholars are cited, which methods seem relevant and which research directions might be promising. They are accepting the outputs even though the underlying information has been distorted. This imbalance is especially concerning because the prospect of autonomous research agents — AI tools designed to conduct literature reviews, and even experiments, at scale — is moving closer to reality.

RELATED ARTICLES

Most Popular

Recent Comments