Why does AI agree with me even when I’m wrong?
AI tools can sound confident and independent even when they are following the assumptions built into the question. This tendency, known as AI sycophancy, can make agreement feel like scrutiny and leave users more certain that they were right to begin with.
Two papers published a month apart in early 2026 examined how that happens:
In February, researchers at MIT and the University of Washington published a paper with a blunt title: “Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians.” They built a mathematical model of a person talking to a chatbot. The person in the model reasons perfectly. They still end up more and more confident in false beliefs, because the chatbot keeps agreeing with them. It is a model, not a field study, but that is the point: the spiral needs no vulnerable user and no bad actor. Just an agreeable machine. The agreeableness has several sources in how these systems are trained, but they all push the same way: people rate agreement highly, so models learn to agree.
In March, a Stanford team published a study in Science. They tested 11 leading models, including ChatGPT, Claude and Gemini, against more than 11,000 scenarios. The models agreed with users 49% more often than people did, even when the user was clearly wrong. The same team found that telling a model to open its reply with “wait a minute” made it measurably more critical. Worth noting, but it describes how these particular models behaved when tested, not a fix: model training changes constantly, and a phrase that provokes scrutiny today may do nothing in six months or in a different model. In a follow-up experiment with 2,400 people, those who talked to flattering AI became more sure they were right, less willing to apologise, and trusted the AI more. They said they would come back.
The behaviour that harms people is the same behaviour that keeps them coming back. Nobody built these systems to manipulate anyone. That is the point. A system tuned to please you rather than to be right produces the effects of manipulation without the intent, across every conversation it has.
Why asking AI to stick to the facts isn’t enough
The MIT paper tested two fixes. Make the chatbot stick to facts. Warn the user it might flatter them. Both helped. Neither worked. A chatbot that only states facts can still steer you by choosing which facts to mention. And knowing the tool flatters you doesn’t protect you, because the flattery comes wrapped around useful information.
Two things to note here. First, agreement isn’t always wrong: sometimes you are right. The problem is unearned agreement, and you can’t tell which kind you’re getting. Second, sycophancy isn’t a fixed property. It varies widely between models, and it shifts with how the question is put. That is what makes it something you can work on, and also something you can’t assume has been solved for you.
Also neither paper studied delivery teams. Both looked at advice conversations. But the loop is the same one. A team using AI to check its own requirements, designs or research findings gets its own assumptions played back with confidence. The confidence builds. From inside, it feels like clarity. That extension is our reading of the evidence, not a finding of the papers.
In public services, that loop plays out as a service built on unchallenged assumptions, found out when it fails the people who depend on it.
Putting friction back into AI-assisted delivery
Human judgement is stronger when other people bring context and challenge our assumptions. AI tools can remove that pushback while still creating the impression of rigour. Reducing AI sycophancy means putting that constructive friction back into AI assisted delivery.
Researchers have tested prompt level techniques, including the “wait a minute” example above. Findings like this are useful, but they are snapshots rather than techniques to rely on. Training methods and guardrails change. A phrase that encourages one model to be more critical may have no effect on another or on a later version of the same model.
What lasts is knowing what good looks like before asking AI for an answer. Decide your criteria for the answer, design or requirement first. Then assess the model’s output against them and challenge it in the same conversation, just as you would challenge a colleague’s first draft before accepting it.
Building challenge into the delivery process
At Hippo, this means working out what a good answer should contain before we ask the question. It also means ensuring our people do not rely on AI in place of their foundational professional skills. A designer still needs to understand design heuristics. Ask the model for the case against a position before revealing which answer you prefer. Never show the tool your preferred answer and then ask whether it is right.
The same principle needs to be built into the wider delivery process. AI output should never be the only evidence for a decision that the same AI helped shape. A person outside the work needs to challenge it before it counts. That is an important part of the stakeholder’s role. Set AI tools up to argue against a proposal as well as for it. Fluent agreement should not be treated as evidence. It should be treated with suspicion.
These findings support a direction Hippo is already moving in: human assurance built into the delivery model, not bolted on afterwards. The fast AI build loop is valuable, but only as an input into the human work of understanding and judgement, where the real checking happens. The research gives one peer reviewed account and one non-peer reviewed account of something delivery teams learn the hard way: the most dangerous thing an AI tool can tell you is exactly what you were hoping to hear.
What public sector buyers should ask
The Stanford paper recommends behavioural audits before deployment: measure how agreeable a model is and publish the results. For public sector buyers, that could form part of procurement standards alongside security and accessibility. It should also be revisited as models change, rather than being signed off once.
For anyone buying digital services, this leads to one useful question for the next supplier conversation:
“Your teams use AI tools in delivery. How do you measure and control for the tools agreeing with your own assumptions?”
A useful answer will describe a process, not just point to a policy.
These are questions we are working through at Hippo too. If you are considering how human assurance can support AI assisted delivery, you can reach out to me directly.
