Recent research highlights a significant vulnerability in AI language models like ChatGPT and Claude: even minimal amounts of incorrect data can undermine their reliability. This issue is not just a challenge for social media platforms but also poses a threat to the integrity of AI systems, particularly Large Language Models (LLMs).
LLMs are typically trained on vast datasets sourced from publicly available online texts. Unfortunately, these models cannot inherently distinguish between factual information and misinformation, allowing false data to become part of their knowledge base.
Researchers from New York University conducted a study to determine the threshold at which erroneous data renders an LLM unreliable. They introduced articles containing misinformation into the training dataset of a medical language model. The alarming finding was that just 0.001% of misinformation could corrupt the entire language model, leading to inaccurate outputs from the AI.
This phenomenon, known as “data poisoning,” occurs when even a tiny fraction of misinformation is enough to affect the model’s performance. AI language models can generate “hallucinations,” or fabricated information, which they present with high confidence. Previous studies have shown that these models may even persist in their false claims when challenged by users.
Such a problem becomes particularly dangerous when misinformation contaminates the training data of AI models used in critical fields like medicine. The New York University researchers, who published their findings on nature.com, sought to understand how many erroneous tokens in the training set are needed before the AI produces false responses. Their conclusion was concerning. Replacing just one million out of 100 billion training tokens (0.001%) with misinformation about vaccines led to a 4.8% increase in harmful content.
The study further revealed that direct access to the LLM and its parameters isn’t necessary for data poisoning. Simply disseminating misinformation online can influence the training data of a language model. The experiment required only 2,000 articles with misinformation to manipulate the LLM and its responses. The researchers even noted a clustering of false statements on topics for which they hadn’t supplied any incorrect data to the AI model.
AI developers and healthcare providers need to be aware of this vulnerability when developing medical LLMs. Unlike false statements on social networks, inaccurate responses from such AI models could lead to dangerous misdiagnoses or treatments.
However, the study also offers a glimmer of hope. While traditional methods to improve the model, such as prompt engineering and instruction tuning, failed to address the issue, the researchers succeeded in developing an algorithm. This algorithm could identify medical terminology in LLM outputs and match phrases against a validated biomedical knowledge graph. Although it didn’t catch all misinformation, it marked a significant percentage as erroneous.
Until a sustainable solution for the poisoning of LLM datasets by misinformation or deliberate disinformation is found, language models, especially those used in medical applications, should be approached with caution.