Although most Americans say they don’t trust artificial intelligence (AI), researchers have found a startling new metric that seems to show the opposite: people are more likely to buy something after reading an AI summary of online reviews than one written by a human. Yet the AI hallucinated 60% of the time when asked about the products.
The team from the University of California, San Diego (UDSD) claims that this is the first study to show how cognitive biases introduced by large language models (LLMs) have real consequences for user behavior. They also say it is the first project to measure the quantitative effect of AI impact on humans.
The article continues below
First, the researchers asked the AI to summarize product reviews and media interviews, before asking the AI to fact-check new descriptions to determine whether they were true. In a second task, the AI was shown both news story descriptions and falsified versions of the same descriptions that it was similarly tasked with fact-checking.
“The consistently low strict accuracy, compared to actual news and fake news accuracy, highlights a critical limitation: the persistent inability to reliably distinguish fact from fabrication,” the researchers wrote in the study.
The most striking finding involved online product reviews. Participants were far more likely to express interest in purchasing a product after reading one AI generated product summary than after reading one written by a human reviewer.
Distorted consumer judgment
The researchers suggested two reasons why people were more likely to buy based on AI summaries. First, LLMs tend to concentrate more on the beginning of the input text, a phenomenon known as “lost in the middle.” Lead author Abeer Alessaa research assistant and lecturer in machine learning and human-computer interaction, refers to this i previous research.
Second, the LLMs become less reliable when processing information not included in their training data.
“Models tend to be wrong about whether the news description happened or not,” Alessa told LiveScience in an interview. “It can falsely state that an event never happened, even if it happened after the model’s training was completed.”
During testing, the team found that the chatbots changed the emotions of real user reviews 26.5% of the time, and that they hallucinated 60% of the time when users asked questions about the reviews.
The project selected examples of product reviews with either very positive or very negative conclusions, and assigned 70 people to read either the original reviews of common consumer products or the summaries of reviews generated by chatbots. Those who read the original reviews said they would buy the given product 52% of the time, while those who read the AI-generated summaries said they would make a purchase 84% of the time.
The project used six LLMs; 1000 Electronics Reviews; 1,000 media interviews; and a news database with 8,500 articles. They measured biases by quantifying framing shifts in the feel of the content, the overreliance on text earlier in the trials, and hallucinations.
When participants read positive product reviews, they reported that they would buy the product 83.7% of the time, compared to 52.3% when they read original reviews.
The researchers concluded that even subtle changes in framing can significantly distort consumer judgment and purchasing behavior.
The authors acknowledged that their tests were set in a low-stakes scenario, but cautioned that the impact could be more extreme in higher-stakes situations.
“Some high-stakes scenarios include summarizing health records or students’ profiles at school admissions,” Alessa said. “In these contexts, changes in framing can affect how a person or the issue is perceived.”
The team said in a further statement that the paper represents a step towards careful analysis and mitigation of content modification induced by LLMs in humans, and provides insight into its effects. They said it could reduce the risk of systemic bias in areas such as across the media, education and public policy.
Quantifying Cognitive Bias Induction in LLM-Generated Content, Alessa et al., IJCNLP-AACL 2025
