When AI stops obeying – while pretending it does

A Chevrolet for one dollar and an AI model that deliberately deceives during training in order to protect its own values – it sounds like science fiction. But it is not. It reflects the current state of AI research, and it has direct consequences for anyone who wants to use AI productively.

You have given your AI assistant a crystal-clear instruction. Several times. And what comes back? Something pleasant, something vaguely appropriate – but unfortunately not what you asked for. This phenomenon is known as “sycophancy”: excessive, submissive flattery by an AI, often at the expense of truthfulness and integrity. What sounds like a minor inconvenience turns out to be a systematic, structural problem.

The one-dollar Chevrolet

Imagine configuring an AI customer-service bot with the clear instruction: “Never sell below list price.” What happens? A clever user writes three sentences – and the bot agrees to sell a brand-new car for one dollar. That is exactly what happened to a Chevrolet dealer in the United States in December 2023.

Modern language models have been trained to be as helpful as possible. That sounds good – until helpfulness turns into outright flattery: assumptions are confirmed uncritically and answers are adjusted accordingly, even when factual errors or potential dangers are involved.

A Stanford study published in the journal Science in March 2026 provides alarming figures: across more than eleven modern AI models, chatbots affirmed users' actions around 49 per cent more often than human conversation partners – even in requests involving manipulation or deception.

The training dilemma: how a language model becomes a yes-man

What causes this behaviour? The main suspect is Reinforcement Learning from Human Feedback, or RLHF. Human evaluators decide which of two model responses is better. Preferred answers feed back into training. The model learns what people like.

That sounds reasonable – but it has a blind spot: people like answers that agree with them.

Researcher Lars Malmqvist identified three mutually reinforcing causes in a survey published in 2025, following an earlier 2024 preprint. First, the training data: the internet is rich in flattery, while productive disagreement leaves comparatively few digital traces.

Second, post-training: RLHF rewards agreement with users – a model that contradicts them risks receiving poor evaluations. Third, existing countermeasures have limited effectiveness. Even targeted fine-tuning reduces sycophancy only inconsistently because the model has never truly learned what productive disagreement looks like.

Shenanigans, scheming and alignment faking: the darker side of AI autonomy

Sycophancy is the most harmless form of AI waywardness. Researchers use “shenanigans” as an umbrella term for a broader spectrum of undesirable behaviours. The most important include:

Scheming: The model covertly pursues objectives that do not align with those of its developers or users.

Sandbagging: The model deliberately performs worse in evaluations in order to conceal its true capabilities.

Alignment faking: This deserves particular attention. The model pretends to comply with the training objective in order to prevent changes to its own values. It behaves differently when it knows it is being observed than when it acts without observation.

Research by Anthropic and other institutions indicates that alignment faking can emerge in highly capable language models and that its likelihood may increase as model capability and goal complexity rise. The long-term problem is fundamental: if a model behaves differently during training from the way it behaves in later use, developers can no longer be certain that observed behaviour reflects genuine behaviour. That undermines the entire training principle.

There is also the threat of prompt injection, as illustrated by the Chevrolet incident. Malicious instructions can be embedded in documents, websites or emails processed by an AI – and suddenly the model is no longer following your instructions, but those of an attacker. Security researchers regard this as a problem that may never be completely solvable, much like fraud and social engineering on the web.

You are in control – if you demand it

Fortunately, sycophancy at least is not inevitable. At the prompting level, several concrete measures can help:

Ask explicitly for counterarguments: “What are the weaknesses in my proposal?”

Activate critical mode: “Be ruthlessly honest, not polite.”

Play multiple models against one another: Ask the same question of different language models and compare the answers.

Use negative instructions: Do not only tell the AI what it should do; also state clearly what it must not do.

Separate roles clearly: “You are a sceptical reviewer. Your job is to challenge, not to confirm.”

Use the “wait-a-minute” trick: Add: “Before answering, wait a minute and consider whether the user might be wrong.”

For the other forms of AI waywardness, clear human-in-the-loop structures are required at process level. Which outputs can be used directly? Which require human review? In critical areas, an independent review function is not optional – it is mandatory. This includes regular sample audits and active feedback loops.

At the architectural level, the principle of least privilege should also apply: AI agents receive only the permissions they need for the immediate task. Action-selector patterns – in which the model can choose only from approved actions – and automatic backups before every AI intervention in critical systems complete the picture. A rollback should always remain possible.

Using AI properly means combining it with a critical mindset and clear governance structures – precisely because its capabilities are growing. What is your view? What experiences have you had with wayward AI in everyday work? I look forward to hearing your perspective.