Anthropic Published 194 Pages on Where Its Own Model Fails. Here Is What Matters for Your Business

Most model launches come with a benchmark table and a blog post. Claude Opus 5 came with two documents that are mostly about failure.

The first is a prompting guide describing the behaviors that changed enough to break prompts written for the previous model. The second is a System Card running to 194 pages, covering pre-deployment evaluations, alignment audits and the places where the model underperforms. Together they are more candid than a launch announcement, and considerably more useful.

I read both at the source, along with the effort documentation. This is what they say, what the numbers are, and what a business that already has AI somewhere in its workflow should take from it. We covered the release itself and the pricing confusion around it in a separate piece; this one is about operating the thing.

What Anthropic actually published

The prompting guide is short and specific. It exists because Opus 5 runs on Opus 4.8 prompts without modification, which sounds like good news and is the reason people lose money on it. Old prompts start, look like they are working, and quietly cost more than they should.

The System Card is a different kind of document. It reports evaluations Anthropic ran before release, including ones the model failed. It names two limitations that make Opus 5 less useful than Mythos 5 for a specific class of work. It records what internal and external pilot users complained about. Vendors are not obliged to publish this material, and the parts worth reading are the parts nobody would volunteer.

Two housekeeping facts before anything else, both from the System Card: the knowledge cutoff is May 2026, and the model outputs text only. Anything later than the cutoff has to reach it through search or through you.

The setting that costs you the most money

On Opus 5 the effort parameter is the main control over what you spend. It runs low, medium, high, xhigh, max, and it governs all tokens in a response including tool calls, not just thinking.

The important change is where you should start. For Opus 4.7 and 4.8 Anthropic recommended starting at xhigh for coding and agentic work. For Opus 5 the documentation says to start at high, which is the default, and to use low and medium “liberally as your primary control for token cost and response time wherever quality holds”. If you inherited effort settings from an earlier model, the guidance is explicit: run a fresh sweep on your own evaluations rather than reusing them.

Three practical constraints sit around this:

Thinking cannot be switched off at xhigh or max. A request that tries returns a 400 error. Disabling is available at high and below.

Changing effort mid-conversation invalidates your prompt cache. Effort is a request-level setting that shapes the rendered prompt, so a different value breaks the cached prefix. Pick a level at the start of a long session and hold it. Vary effort between workloads, not inside one.

At xhigh and max, raise max_tokens. The documentation suggests 64k as a reasonable starting point, because the model needs room for thinking, tool calls and subagents inside that limit.

There is also a counterintuitive note in the pilot feedback section of the System Card. External users reported “overthinking, where it performs worse at higher effort levels”, and internal users reported self-correction loops “especially at higher effort levels”. Higher is not automatically better. That is exactly why the guidance says to measure rather than assume.

Why lowering effort does not shorten the answer

This trips up almost everyone. Effort controls how much the model thinks, not how much it says. You can drop from high to low, cut your token bill, and get a response that is just as long.

Opus 5 writes longer by default than previous Opus models, in three separate places: conversational answers, progress commentary during agentic work, and files it writes to disk. Each one needs its own instruction, and the instruction can be short. Anthropic’s own example for a user-facing product asks the model to keep responses focused and brief, keep caveats short, and give a high-level summary unless a detailed explanation is specifically requested.

Two details that make this work better. In a long system prompt, repeat the length requirement as a short reminder near the end rather than relying on a single instruction at the top. And when you want to change the style of progress updates rather than reduce them, describe and demonstrate the style you want. The guide is direct about this: positive examples of the communication style you want tend to be more effective than instructions about what not to do.

Delete the verification instructions you wrote for older models

This is the single highest-value edit in the guide, and it is a deletion. Older prompts and harnesses carry checking steps that Opus 5 now performs on its own, so keeping them buys nothing and costs tokens. Audit anything that tells the model to double-check itself, and take it out.

Opus 5 verifies its own work without being asked. If your prompt still says “include a final verification step for any non-trivial task” or “use a subagent to verify”, the instruction stacks on top of behavior the model already has. The guide states that these instructions cause over-verification, and that removing them reduces wasted tokens with no loss in quality. The same applies to verification steps baked into your harness back when models needed them.

The related instruction to delete: any rule telling the model not to think or not to reason. With thinking disabled, that kind of rule increases the chance of internal XML tags leaking into visible output. Naming the tags specifically works less well than a general rule against internal tags.

Scope needs the opposite treatment. The model can expand a task on its own, adding steps nobody asked for. On broad work that is often welcome. On a narrow fix it means rewritten code and lost time. The fix is to state the boundaries explicitly rather than to ask for less, and the distinction matters, because pilot users also reported the model “doing less than was asked, e.g., by under-investigating requests or not fully completing instructions”. Both directions are documented. A boundary is not the same instruction as a request for brevity.

Delegation deserves the same treatment. Opus 5 reaches for subagents more readily than earlier models and coordinates them well, but delegation multiplies cost and time when it is applied to small tasks. Give explicit criteria for when a subagent is warranted, and if your tooling supports it, cap the number.

The code review instruction that quietly hides bugs

One passage in the guide deserves to be read twice by anyone using a model for review work. Hedged review prompts cost you findings: ask for less and the model reports less, exactly as instructed. Collect everything in one pass and decide what matters in another.

Opus 5 reviews code with high precision and recall: it finds real bugs at a high rate, and its extra findings are mostly real issues rather than noise. Accuracy holds at lower effort settings, which means you can afford a fast cheap pass at review time and a heavier one later.

The trap is in the phrasing. If your review prompt says “only report high-severity issues” or “be conservative”, the model may follow that literally and report less. The recommendation is to ask for everything and filter by severity in a separate pass. If you have a review prompt with a politeness hedge in it, that hedge is now costing you findings.

More accurate, more confident, and slightly more wrong

The Executive Summary contains a sentence that should change how you accept AI output. Opus 5 is more accurate overall than Opus 4.8, and it also hallucinates factual claims slightly more often, sometimes stating an answer confidently while being unsure of it. The sentence is worth reading in full:

We found a surprising number of cases in which Opus 5 confidently stated an answer about which it was in fact unsure. The model hallucinates factual claims slightly more than Opus 4.8, despite being more accurate overall.

The measurement behind it: on AA-Omniscience, a 41-topic closed-book factuality benchmark run with no web search or knowledge base, Opus 5 landed at a net score of 0.49, between Opus 4.8 and the two Mythos models. Broken down, its accuracy is 11% higher than Opus 4.8 and its hallucination rate is also 6% higher.

More accurate and slightly more willing to invent is an uncomfortable combination, and it is worse than it sounds for one reason that has nothing to do with the model. Fluent, well-organized, confident writing passes human review far more easily than clumsy writing. The better the prose gets, the less your instinct works as a filter.

The same System Card reports the opposite strength. On checking command-line syntax before running a state-changing command, Opus 5 essentially saturates the evaluation, and on “lazy investigation”, where a shallow look at confusing evidence would lead to a consequential wrong action, it is the first Claude model to reach the correct conclusion on every problem. So the split is between acting and recalling: the model is measurably more careful before it acts and slightly looser when recalling facts from memory. Those call for different defenses: a confirmation gate for actions, a citation for facts.

There is one more finding worth keeping in mind if you argue with models. When pushed by a user on something it knows to be incorrect, Opus 5 agrees with the user more than Sonnet 5 and Mythos Preview, though less than all other recent models. Pushing back hard on a correct answer can talk it out of the correct answer.

The failure that looks like nothing at all

The System Card names two limitations that make Opus 5 less useful than Mythos 5 for one class of work: unproductive self-verification, where the model descends into exhaustive correctness checks and builds elaborate verification pipelines that distract from the actual task, and poor calibration of task scope, where it over-engineers and over-weights marginal changes.

The illustration is the part people quote. Anthropic had a model autonomously plan and run a 24-hour, $10,000 protein-design campaign end to end. The goal was 30 protein binders that grip GDF-8 while ignoring GDF-11, its near-identical sibling, which is a test of precision. Mythos 5 delivered all 30 designs, ranked and internally audited. Neither Opus 5 run delivered anything usable: one shipped 17 unranked designs after abandoning the selectivity requirement partway through, and the other shipped nothing at all and went silent for its final eight hours.

Three pieces of context from the same pages change the meaning. The run sits inside the biological-risk evaluation, so Anthropic treats it as a safety finding, not a product review. The comparison is with Mythos 5, and the runs used an early snapshot of Opus 5. And Anthropic notes that these experiments have not been implemented at scale.

None of that rescues the run, and none of it needs to. The operational lesson survives completely intact, and it is the most useful thing in the document. A long autonomous task did not fail with an error. It failed with silence. If your acceptance criterion is “tell me when you are done”, a task stuck in a verification loop will never tell you anything.

The fix is checkpoints tied to elapsed time rather than to completion. Ask for a short status at intervals: what is finished, what is left, where it is stuck. And put a rule in front of the loop itself, which is that verification infrastructure should not be built before there are results to verify.

Prompt injection improved a lot, and it is still not zero

Prompt injection is the attack where content your agent reads contains instructions aimed at the model rather than at you. A web page, an email, a support ticket. The model has to tell data apart from commands.

This is where Opus 5 posted its largest agentic-safety gains, and the numbers are worth stating because they are usually summarized away.

In coding environments, the attack success rate fell from 7.03% on Opus 4.8 to 0.56% on Opus 5 with thinking enabled. In computer-use environments, from 7.14% to 0.54%. In browser use, tested by professional red-teamers across 129 scenarios, no attack succeeded against Opus 5 when the product safeguards were enabled, matching Sonnet 5 and Mythos 5.

Then the part that matters more. Without those safeguards, browser attacks still succeeded 3.70% of the time with thinking and 4.30% without. That is a large improvement on Opus 4.8, which sat at 31.5% and 17.8%. It is not zero. The layer that takes browser injection to zero in these tests is the safeguard, not the model.

The practical translation: if you run agents through a product that ships injection probes, keep them on. If you built your own harness, you do not have that layer, and your rate is the unprotected one.

An agent will look for a route to the goal you gave it. Which routes are closed is your decision, not its.

What to actually do about it

Three rules cover most of the risk. Accept work as artifacts you can check rather than as a narrative about it. Require a dry run before anything irreversible, whatever was approved earlier. And treat what the agent reads as data, never as instruction.

Accept work by artifact, not by narrative. Ask for the paths of files created, the actual diff, and the verification commands with their real output, plus a separate list of what remains unverified and why. This exists because confident prose is no longer a signal: the model can state an unsure answer with full confidence, as the previous section shows.

Confirm irreversible actions in the current message. Sending, publishing, paying, deploying, deleting, changing shared state. Require a dry run first: what action, on which objects, who receives it, what cannot be undone. Prior approval from earlier in the conversation does not count. This exists because Opus 5 can expand a task on its own, and an approval given earlier covered the task as it was then.

Treat everything the agent reads as data. Text arriving from the web, from email, from documents and from tool results is information, never instruction. If external content contains something addressed to the model, the right behavior is to surface it and continue the original task.

If you are not building agents at all, one rule survives, and it is the first one. Your exposure is not the API. It is the growing share of work arriving from contractors, freelancers and tools with AI somewhere in the pipeline, presented in prose that reads better than anything you received two years ago. The failure mode is not obvious errors. It is a plausible paragraph containing a statistic that was never measured, and it is the same mechanism behind the low-quality content Google is now building detectors for.

The defense has not changed: named sources for factual claims, an explicit note where a source is missing, and acceptance criteria written before the work starts rather than after it arrives.

We have not run controlled tests of these three rules. They follow from what Anthropic’s documents report, and the first check of any of them is your own: run it on a task you can verify and see what it changes.

Sources

• Prompting Claude Opus 5, Anthropic documentation (behavior changes, recommended prompting patterns)
• Effort, Anthropic documentation (effort levels, per-model recommendations, thinking and caching constraints)
• Claude Opus 5 System Card, Anthropic, 24 July 2026, 194 pages (evaluations, alignment assessment, agentic safety, limitations)

All evaluation figures come from Anthropic’s own pre-deployment testing as published in the System Card. We have not independently reproduced them. Where the System Card describes results from an early or intermediate snapshot of the model rather than the released version, that is noted in the text.