AI conversations are often presented as a way to avoid effort. Type a question, receive an answer and move on. But a Stanford study of real conversations with Anthropic’s Claude suggests that useful human-machine work can be more demanding. People frequently question, refine and correct what the chatbot produces. Disagreement is not necessarily a sign that the interaction has failed. It may be the moment when the user starts thinking hardest.
The research analysed 249,834 anonymised conversations and looked at the nature of the task, how much control the person retained and what happened when the exchange became difficult. It found that a substantial share involved consequential work, rather than casual experimentation. That is an intriguing change of emphasis: the value of AI may lie partly in the discussion surrounding the answer, not just the answer itself.
AI conversations are not always passive requests
The Stanford researchers used an analytical system made available through Anthropic to examine patterns in Claude conversations while protecting individual privacy. The team asked whether people were completing trivial or important tasks, whether they stayed in charge of the work and where the exchange encountered friction. It was an attempt to understand actual interaction rather than rely solely on predictions about how people might use the technology.
According to the researchers, 56 per cent of conversations were classed as consequential or more significant. That classification reflects the study’s analytical criteria; it does not mean an independent observer verified that every result changed a real-world decision. The paper examines activity across professional and advisory contexts where a mistaken recommendation might matter, as well as conversations with much lower stakes.
The team also reported that conversations became longer as the importance of the work increased. This is a useful warning against treating shorter exchanges as automatically superior. A person asking a chatbot to spell-check an invitation may need a quick response. Someone planning a complex project or checking a legal interpretation may be safer if they ask several questions, challenge assumptions and verify the evidence elsewhere.
The paper has been described by Stanford as accepted by Nature Human Behaviour. That is not the same as treating every interpretation in a university news article as an already established scientific consensus. The work is an important dataset with specific methods and limits. A second source, the research paper available through alphaXiv, gives the authors’ own description of what was measured.
What researchers mean by friction
Friction sounds like an obstacle to efficiency. In this study, it refers to points where the person’s goal and the AI’s response do not line up smoothly. The model may misunderstand the task, make an error or offer a solution the user rejects. The user may clarify, correct, dispute or ask for a different approach. Some of these exchanges lead nowhere; others become useful moments of learning or refinement.
The researchers found friction in roughly half of the conversations. That does not establish that half of the interactions became better as a result. Some friction is simply frustration. A wrong answer can waste time, especially when the person lacks enough background knowledge to recognise it. The interesting point is that users often remain active rather than treating the first response as the final authority.
In ordinary working life, useful friction is familiar. An editor challenges a headline, an accountant questions a figure or a colleague asks why a proposed solution will work. These exchanges take time, but they may prevent a poor decision. AI can imitate part of that discussion by offering counterarguments or alternatives. It cannot supply the independent accountability, lived experience or verified facts that another professional might bring.
The difference between productive and unproductive disagreement depends on evidence. If the chatbot challenges a claim and explains a genuine weakness, the exchange can be useful. If it confidently invents a rule or misunderstands the context, the extra conversation may only deepen confusion. Users need a way of testing the claim outside the conversation when the stakes are significant.
People are still doing much of the steering
The paper reports that human-led collaboration dominates the interactions examined. People often choose the objective, decide how much to delegate and revise responses through successive prompts. That challenges the popular image of someone pressing a button and surrendering an entire decision to software. At least in this sample, many users were negotiating the work as it developed.
The authors also found that AI-supported teaching or learning appeared in a substantial share of conversations, with a reported figure of 67 per cent under their classification. That does not prove the users learnt lasting skills or that the model was a good teacher. It means the conversations showed features the researchers interpreted as teaching or instruction. Learning outcomes require separate testing, preferably with independent measures rather than the apparent success of a chat.
What matters is the behaviour the system encourages. A tool that always offers a complete answer may make it easy for someone to avoid confronting a difficult idea. A tool that explains an assumption and asks a useful follow-up may help the person understand the work. Yet neither approach is appropriate for every task. Someone requesting a train timetable does not need a philosophical debate; someone interpreting a disputed contract might.
The user’s agency can also shift within a conversation. They may ask for ideas, take one suggestion, reject another and then demand a clearer explanation. The model’s role is not fixed by its name. This is why a serious account of AI at work should examine how people interact with it rather than categorising every use as either automation or assistance.
An apparent partner is still a machine
Language can give a misleading impression of shared judgement. A chatbot can say it agrees, disagrees or has reconsidered a position, but those statements should not be confused with human responsibility. It generates text according to its design and context. The person still needs to own decisions, particularly where law, safety, money or another person’s wellbeing is involved.
The same is true of compliments. A model may praise a weak proposal because it is tuned to be polite or because the prompt rewards agreement. A useful challenge should be specific: identify an unsupported figure, suggest a missing alternative or explain an inconsistency. Saying a plan is brilliant contributes little. The test is whether the conversation makes the reasoning more accurate and the decision more defensible.
A practical method is to ask the assistant for the strongest reasonable objection to a proposal, then examine the objection without assuming it is true. Ask which facts would decide the issue. If the answer cites a law, price or study, check the original source. This gives the user more control than either accepting the first response or demanding that the chatbot always be argumentative.
Human expertise remains especially important when a model cannot observe the situation directly. A manager may know a supplier’s history, an engineer may recognise a physical hazard and a teacher may understand why a pupil is struggling. Those facts can be decisive even when they are absent from the text conversation. LiveAIWire has explored the role of human judgement in AI-assisted engineering work, where an attractive recommendation still needs experienced scrutiny.
Where the study cannot take us
This analysis was based on conversations with Claude, not a random sample of every person who has ever used AI. People using that service, in the period studied, may differ from users of other products or from people who never open a chatbot. The dataset also contains privacy-preserving analytical classifications rather than researchers personally observing the full consequences of every exchange.
The sample is valuable because it is large and grounded in real interactions. Yet volume does not eliminate selection bias, and an automated label does not prove that a user retained genuine control. Some people may appear active because they repeatedly correct errors they would not have encountered without the system. A long conversation is not automatically a productive one.
Nor does the study establish that AI increases human competence over time. That question would require comparing skills before and after use, or observing longer-term work outcomes. The researchers themselves distinguish between the appearance of teaching and whether capability is actually built. It is a central issue for a world in which young professionals may increasingly rely on systems they do not fully understand.
LiveAIWire’s discussion of German apprentices using AI beyond the tasks they were trained for shows why delegation and skill development should be considered together. A tool can expand what someone attempts while leaving unresolved questions about their understanding. The risk is not simply that work becomes automated. It is that people may lose sight of where their own responsibility begins and ends.
What useful disagreement looks like in practice
Start with a concrete objective and enough context for the assistant to understand it. Ask for its proposed reasoning, then challenge the assumptions most likely to matter. If the response includes a claim that can be checked, seek the underlying evidence. If it offers a recommendation, ask what circumstances would make a different choice sensible. That process keeps the person involved without turning every task into an elaborate interrogation.
When the model is plainly wrong, continuing the conversation may not help. The right action can be to stop, check an authoritative source or speak with someone qualified. Productive friction is not an obligation to argue indefinitely. It is an opportunity to improve the work where a real question remains open. Learning when to leave the chatbot is part of sensible use.
There is also a management consequence. Organisations should not assess AI solely by the number of minutes saved on a first draft. They should examine mistakes avoided, quality improvements and whether people understand the decisions they approve. LiveAIWire has reported how faster AI-assisted work can create additional responsibilities, an example of why time and value do not always move together.
The more interesting measure of AI usefulness
The Stanford study challenges the idea that seamless agreement is the ideal user experience. Some collaboration is effective because someone notices a problem, challenges the answer and improves the result. A model that allows this kind of exchange may be more useful than one that simply sounds reassuring. The evidence does not justify celebrating every disagreement, but it does suggest that friction deserves closer attention.
The next research question is whether these exchanges improve skills and outcomes beyond the conversation itself. Do people make better decisions after challenging an AI? Are they more able to perform the task unaided? Which kinds of correction actually improve accuracy? The current analysis opens those questions without settling them.
For users, the lesson is more immediate. A good chatbot session need not feel effortless. If the answer matters, it may be worth asking an awkward question, rejecting an easy assumption or checking the source before moving on. The most valuable contribution may be the thought that happens after the model has spoken.
About the Author
Stuart Kerr is Technology Correspondent at LiveAIWire, covering artificial intelligence, cybersecurity and the social impact of emerging technology. LiveAIWire is an independent, human-led technology publication using AI-assisted research, editorial production and original AI-assisted editorial illustrations under his direction.