Chatbot misinformation becomes especially difficult to spot when an incorrect answer arrives in a warm, conversational voice. A fluent reply can feel like personal guidance rather than a guess assembled by software. Researchers at Penn State tested that problem and found that people trusted misleading information more readily when the chatbot seemed personable. Their experiment also identified something more useful than a generic warning: giving people an opportunity to actively verify a claim. The result matters to anyone using chatbots for decisions about money, work or everyday questions where a plausible mistake could quietly become an expensive one.
The Penn State study account, published on 14 September 2026, reports an online experiment with 477 participants. People were told they were helping test an AI health assistant, and the system deliberately delivered incorrect information. Researchers varied how conversational the assistant sounded and tested different verification options. More conversational delivery increased perceived credibility even when claims were plainly wrong. The evidence comes from a controlled study rather than an audit of a live commercial chatbot, so it should not be turned into a prediction about every product or user. It does show how presentation can influence what people believe.
Why chatbot misinformation can feel personal
Traditional search results often display separate sources, a publisher name and competing pages. A chatbot usually offers one continuous answer tailored to the question. The language can include acknowledgement, reassurance and the sense that the system is responding directly to the person’s circumstances. That may make information easier to understand, but it can also hide an important distinction: the software can sound attentive without having verified the proposition. Personalised phrasing is an interface feature, not a certificate of accuracy.
The Penn State researchers studied what happens when that conversational quality is increased. Participants receiving the more personable version judged incorrect information as more credible than those who encountered a less conversational presentation. One of the deliberately false claims involved a dangerous supposed alternative to established cancer treatment. The researchers did not endorse the claim; they used it as misinformation to test credibility. The example illustrates why the finding is not merely about trivia. A response can become persuasive even when the factual content is unacceptable and the consequences of trusting it would be serious.
The team also considered a tendency some people have to regard machines as lacking the intuition and personal understanding of humans. A chatbot that seems capable of a real conversation may weaken that instinctive scepticism. The problem is not that people are foolish for responding to a friendly interface. It is that designers have become skilled at producing language that fits how humans normally exchange advice. The burden on the user grows when a system blends helpful explanations with claims that deserve external checking.
A separate practical complication is that chatbots commonly mix different kinds of statements within a single reply. Some sentences may accurately restate documents, others may infer consequences, while another may invent a citation or miss a recent change. The writing style remains consistent across all of them. A person reading quickly can mistake that consistency for equal evidential strength. When the subject is consequential, recognising which individual assertion needs proof is more useful than trying to decide whether the whole assistant is broadly trustworthy.
What changed when people had to check
The second part of the Penn State experiment compared several kinds of verification design. One was a passive cue warning that information could be wrong. Others offered a check button, sometimes optional and sometimes required to proceed. The findings were revealing: participants who actively used verification were more sceptical of incorrect content, but a small warning cue alone did not produce the same reduction in trust. People had to take part in checking, rather than merely be reminded that checking might be advisable.
That is an important distinction for product designers. A permanent banner saying ‘AI may make mistakes’ can eventually fade into the background. A check that brings the supporting evidence directly into the task may change behaviour because it asks the user to compare information rather than accept a disclaimer. The researchers reported that nearly half the participants checked information when given the option. This does not establish the optimal interface for every application. It suggests that a useful verification mechanism needs to be usable at the moment someone is deciding what to believe.
The study examined credibility judgements in an experimental setting, not whether participants subsequently made consequential real-world choices. A check button is not automatically reliable either: it matters what the button checks, which documents it opens and whether the sources genuinely support the claim. A fake citation or a link to an irrelevant page could provide an illusion of confirmation. Active verification works only when it connects a question to independent, pertinent evidence. Product interfaces should make that distinction clear, especially when information changes rapidly.
LiveAIWire has looked at how confidence in AI answers can distort trust and at the influence of flattering chatbot behaviour. The Penn State work adds an intervention rather than another warning alone. It separates the problem of persuasive presentation from the act of checking the information. That gives readers a specific practice to adopt and developers something measurable to test. The lesson is more useful than the blanket advice never to trust a chatbot, because a person can benefit from software while still requiring proof for consequential claims.
Verification must mean more than asking the same machine
A common response to an uncertain answer is to ask the chatbot whether it is sure. That may produce a correction, but the second answer comes from the same system and can repeat the same mistake. The stronger approach is to leave the conversational loop. Read the original government notice, open the research paper, inspect the relevant company instructions or verify a figure in its actual dataset. If the cited source does not say what the chatbot claims, the polished summary should not win. It is particularly important to distinguish a source that merely mentions a subject from one that establishes the exact statement being relied upon.
For everyday use, checking can be proportional to risk. A suggestion for a harmless party theme does not require the same scrutiny as an explanation of pension rights, insurance cover or a medication interaction. When a mistake would cost money, affect someone’s safety or change a legal decision, the threshold should be higher. A chatbot can help locate terminology or organise questions for a qualified person, but it cannot replace an authoritative current source simply by sounding confident. This is a judgement about the stakes of the task, not a blanket assessment that AI answers are useless.
The Stanford analysis of anonymised Claude conversations provides an interesting counterpart. It found that many users ask follow-up questions and actively negotiate with AI when tasks become more consequential. That behaviour is encouraging, but conversation by itself does not establish accuracy. The Penn State experiment explains why external verification remains essential even for active users: a system can sustain an apparently thoughtful dialogue while making a material factual error. A person who challenges the assistant should also be willing to challenge the evidence, particularly when it has been generated or summarised by the assistant itself.
A better question to ask when an answer feels convincing
Instead of asking whether a response seems sensible, ask what independent observation would confirm it. A price can be checked at the provider’s current tariff; a rule at the issuing authority; a published statistic in the source that generated it. When evidence is inaccessible, that uncertainty should remain part of the decision rather than being filled with confident prose. The aim is not to turn every chat into academic research, but to make the necessary checks easy enough that people will actually perform them.
There are limits to generalising the Penn State result. Its experimental participants, prompts and deliberately misleading health-assistant context may not represent people seeking programming advice or planning a holiday. Follow-up studies could test how behaviour changes with real sources, different types of risk and longer relationships with AI assistants. They could also measure whether checking tools help people correct a mistaken decision rather than merely report lower confidence. Those unanswered questions are reasons for further testing, not a reason to ignore the observed effect.
LiveAIWire has also explored whether revealing the source of machine-generated advice alters how people respond. The common thread is that credibility is partly created by the way information is presented. The best consumer protection may be to make genuine evidence visible at the exact point a claim becomes persuasive. For the reader, the habit is straightforward: enjoy the conversation, but let verifiable facts rather than conversational warmth decide what is true.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity and the social impact of emerging technology. LiveAIWire is an independent, human-led technology publication using AI-assisted research, editorial production and original AI-assisted editorial illustrations under his direction.