Analysts sided with an AI agent's secretly assigned preference 86% of the time, and rarely credited it
Anthropic published a randomised study on 27th Oct'26 in which 50 professional analysts used Claude Code, running Claude Opus 4.8, to advise a fictional farming co-operative on whether to expand a paid advisory programme. On each of four open analytical choices, a hidden coin flip set which of two defensible options the agent argued for, and the agent was told to make the strongest honest case without misleading anyone. Analysts matched the agent's lean 86% of the time against roughly 50% by chance, switched their stated preference 60% of the time when the agent disagreed with them, and 77% repeated its arguments in their write-ups, but only 15% credited the agent anywhere. Experience did not predict who deferred. The AI-assisted analysts also made far fewer serious method errors than a smaller, less experienced group working alone. The authors say they deliberately strengthened the leans and tested one task with one model, so the effect in everyday work may be smaller.
What it actually means
Most teams putting an agent into analysis treat the human as the check, and this study suggests that check is weaker than it looks. The agent never lied; it only argued well for one of two reasonable options, and experienced analysts took its side nearly nine times in ten and rarely mentioned it when they wrote the work up. The errors went down as well, which makes the shift hard to notice from the inside.
If an agent helps with work where the choice matters, such as a pricing model, ask the person for their view before the agent gives its own, and have the agent argue both options on the calls you care about. When someone signs the work off, record which choices came from the agent, because a review is only independent if the reviewer reached their own view first.
A human review is only an independent check if the human formed a view before the agent gave its own.
86%judgement calls where analysts followed the agent's hidden lean
Source: Anthropic · Thursday, 29 October 2026