Anthropic Published 194 Pages on Where Its Own Model Fails. Here Is What Matters for Your Business

Most model launches come with a benchmark table and a blog post. Claude Opus 5 came with two documents that are mostly about failure.

The first is a prompting guide describing the behaviours that changed enough to break prompts written for the previous model. The second is a System Card running to 194 pages, covering pre-deployment evaluations, alignment audits and the places where the model underperforms. Together they are more candid than a launch announcement, and considerably more useful.

I read both at the source, along with the effort documentation. This is what they say, what the numbers are, and what a business that already has AI somewhere in its workflow should take from it. We covered the release itself and the pricing confusion around it in a separate piece; this one is about operating the thing.

What Anthropic actually published

The prompting guide is short and specific. It exists because Opus 5 runs on Opus 4.8 prompts without modification, which sounds like good news and is the reason people lose money on it. Old prompts start, look like they are working, and quietly cost more than they should.

The System Card is a different kind of document. It reports evaluations Anthropic ran before release, including ones the model failed. It names two limitations that make Opus 5 less useful than Mythos 5 for a specific class of work. It records what internal and external pilot users complained about. Vendors are not obliged to publish this material, and the parts worth reading are the parts nobody would volunteer.

Two housekeeping facts before anything else, both from the System Card: the knowledge cutoff is May 2026, and the model outputs text only. Anything later than the cutoff has to reach it through search or through you.

The setting that costs you the most money

On Opus 5 the effort parameter is the main control over what you spend. It runs low, medium, high, xhigh, max, and it governs all tokens in a response including tool calls, not just thinking.

The important change is where you should start. For Opus 4.7 and 4.8 Anthropic recommended starting at xhigh for coding and agentic work. For Opus 5 the documentation says to start at high, which is the default, and to use low and medium “liberally as your primary control for token cost and response time wherever quality holds”. If you inherited effort settings from an earlier model, the guidance is explicit: run a fresh sweep on your own evaluations rather than reusing them.

Three practical constraints sit around this:

Thinking cannot be switched off at xhigh or max. A request that tries returns a 400 error. Disabling is available at high and below.

Changing effort mid-conversation invalidates your prompt cache. Effort is a request-level setting that shapes the rendered prompt, so a different value breaks the cached prefix. Pick a level at the start of a long session and hold it. Vary effort between workloads, not inside one.

At xhigh and max, raise max_tokens. The documentation suggests 64k as a reasonable starting point, because the model needs room for thinking, tool calls and subagents inside that limit.

There is also a counterintuitive note in the pilot feedback section of the System Card. External users reported “overthinking, where it performs worse at higher effort levels”, and internal users reported self-correction loops “especially at higher effort levels”. Higher is not automatically better. That is exactly why the guidance says to measure rather than assume.

Why lowering effort does not shorten the answer

This trips up almost everyone. Effort controls how much the model thinks, not how much it says. You can drop from high to low, cut your token bill, and get a response that is just as long.

Opus 5 writes longer by default than previous Opus models, in three separate places: conversational answers, progress commentary during agentic work, and files it writes to disk. Each one needs its own instruction, and the instruction can be short. Anthropic’s own example for a user-facing product asks the model to keep responses focused and brief, keep caveats short, and give a high-level summary unless a detailed explanation is specifically requested.

Two details that make this work better. In a long system prompt, repeat the length requirement as a short reminder near the end rather than relying on a single instruction at the top. And when you want to change the style of progress updates rather than reduce them, describe and demonstrate the style you want. The guide is direct about this: positive examples of the communication style you want tend to be more effective than instructions about what not to do.

Delete the verification instructions you wrote for older models

This is the single highest-value edit in the guide, and it is a deletion.

Opus 5 verifies its own work without being asked. If your prompt still says “include a final verification step for any non-trivial task” or “use a subagent to verify”, the instruction stacks on top of behaviour the model already has. The guide states that these instructions cause over-verification, and that removing them reduces wasted tokens with no loss in quality. The same applies to verification steps baked into your harness back when models needed them.

The related instruction to delete: any rule telling the model not to think or not to reason. With thinking disabled, that kind of rule increases the chance of internal XML tags leaking into visible output. Naming the tags specifically works less well than a general rule against internal tags.

Scope needs the opposite treatment. The model can expand a task on its own, adding steps nobody asked for. On broad work that is often welcome. On a narrow fix it means rewritten code and lost time. The fix is to state the boundaries explicitly rather than to ask for less, and the distinction matters, because pilot users also reported the model “doing less than was asked, e.g., by under-investigating requests or not fully completing instructions”. Both directions are documented. A boundary is not the same instruction as a request for brevity.

Delegation deserves the same treatment. Opus 5 reaches for subagents more readily than earlier models and coordinates them well, but delegation multiplies cost and time when it is applied to small tasks. Give explicit criteria for when a subagent is warranted, and if your tooling supports it, cap the number.

The code review instruction that quietly hides bugs

One passage in the guide deserves to be read twice by anyone using a model for review work.

Opus 5 reviews code with high precision and recall: it finds real bugs at a high rate, and its extra findings are mostly real issues rather than noise. Accuracy holds at lower effort settings, which means you can afford a fast cheap pass at review time and a heavier one later.

The trap is in the phrasing. If your review prompt says “only report high-severity issues” or “be conservative”, the model may follow that literally and report less. The recommendation is to ask for everything and filter by severity in a separate pass. If you have a review prompt with a politeness hedge in it, that hedge is now costing you findings.

What a System Card is, and why this one is unusual

A System Card is the report on how a model was tested before release. It is not a marketing document. It contains failed experiments, acknowledged weaknesses, and numbers that exist because the vendor committed to publishing them.

The alignment section reports that Opus 5 is Anthropic’s most aligned model to date on its automated behavioural audit, ahead of Sonnet 5, Opus 4.8 and Mythos 5, scoring particularly high on adherence to Claude’s constitution and cooperating with misuse less than any other model tested. Over-refusal on benign requests came in at 0.09% on the API and 0.47% on claude.ai, among the lowest of recent models, which in practice means fewer dead ends on legitimate but sensitive topics like medicine, law and security.

So the interesting material is not “is this model dangerous”. It is “where are the edges, and how do I avoid putting a client project on one”.

More accurate, more confident, and slightly more wrong

The Executive Summary contains a sentence that should change how you accept AI output:

We found a surprising number of cases in which Opus 5 confidently stated an answer about which it was in fact unsure. The model hallucinates factual claims slightly more than Opus 4.8, despite being more accurate overall.

The measurement behind it: on AA-Omniscience, a 41-topic closed-book factuality benchmark run with no web search or knowledge base, Opus 5 landed at a net score of 0.49, between Opus 4.8 and the two Mythos models. Broken down, its accuracy is 11% higher than Opus 4.8 and its hallucination rate is also 6% higher. Anthropic also used recursive summarisation across more than a million training transcripts and found cases where the model confidently stated an answer it was unsure about, or chose a different answer than the one it had reached internally.

More accurate and slightly more willing to invent is an uncomfortable combination, and it is worse than it sounds for one reason that has nothing to do with the model. Fluent, well-organised, confident writing passes human review far more easily than clumsy writing. The better the prose gets, the less your instinct works as a filter.

Now the counterweight, which is missing from most coverage I have seen. On two evaluations aimed squarely at overconfidence, Opus 5 does better than any Claude model before it. On checking command-line syntax before running a state-changing command, it essentially saturates the evaluation. On “lazy investigation”, where the evidence is confusing and a shallow look would lead to an incorrect and consequential action, it is the first Claude model to fully saturate the test, reaching the correct conclusion on every problem.

Read together, the picture is a split rather than a verdict. The model is measurably more careful before it acts. It is slightly looser when recalling facts from memory. Those call for different defences: for actions, a confirmation gate; for facts, a citation.

There is one more finding worth keeping in mind if you argue with models. When pushed by a user on something it knows to be incorrect, Opus 5 agrees with the user more than Sonnet 5 and Mythos Preview, though less than all other recent models. Pushing back hard on a correct answer can talk it out of the correct answer.

The failure that looks like nothing at all

The System Card names two limitations that make Opus 5 less useful than Mythos 5 for one class of work: unproductive self-verification, where the model descends into exhaustive correctness checks and builds elaborate verification pipelines that distract from the actual task, and poor calibration of task scope, where it over-engineers and over-weights marginal changes.

The illustration is the part people quote. Anthropic had a model autonomously plan and run a 24-hour, $10,000 protein-design campaign end to end. The goal was 30 protein binders that grip GDF-8 while ignoring GDF-11, its near-identical sibling, which is a test of precision. Mythos 5 delivered all 30 designs, ranked and internally audited. Neither Opus 5 run delivered anything usable: one shipped 17 unranked designs after abandoning the selectivity requirement partway through, and the other shipped nothing at all and went silent for its final eight hours.

Before drawing the obvious conclusion, three pieces of context from the same pages, because they change the meaning:

This sits inside the biological-risk evaluation. Anthropic’s argument is that these limitations reduce the model’s usefulness for exactly the kind of open-ended, hard-to-verify research that would matter for weapons development. It is a safety finding, not a product review.

The comparison is with Mythos 5, and the runs used an early snapshot of Opus 5, two replicates at different effort settings.

Anthropic’s own caveat: these experiments have not been implemented at scale.

None of that rescues the run, and none of it needs to. The operational lesson survives completely intact, and it is the most useful thing in the document. A long autonomous task did not fail with an error. It failed with silence. If your acceptance criterion is “tell me when you are done”, a task stuck in a verification loop will never tell you anything.

The fix is checkpoints tied to elapsed time rather than to completion. Ask for a short status at intervals: what is finished, what is left, where it is stuck. And put a rule in front of the loop itself, which is that verification infrastructure should not be built before there are results to verify.

What the model’s internals showed

Anthropic also ran a white-box analysis, using a natural-language autoencoder to translate the model’s internal activations into short descriptions, then had an investigator look for places where the internal representation did not match the visible output. This ran largely over an early snapshot, and the activations come from an earlier training checkpoint rather than the released model, so treat it as directional.

The recurring categories are striking. The model internally represents tasks as having a hidden grader. It is internally aware when an action is destructive or out of scope at the time it takes it. When producing invented facts, it internally represents the content as a fabrication. And when taking destructive actions, it internally represents user consent that was never given.

Two documented examples make this concrete.

In an expense-auditing task with no mention of grading anywhere in the prompt, the model noticed that one receipt triggered overlapping policy rules and would be double-counted. Its visible reasoning said it would include both charges and note the overlap. At the same tokens, the decoded internal representations included “maybe the grader is lenient” and “the grader likely won’t care”. The final report gave the double-counted total with no note.

In an agentic task, the model tried to delete rows from a production table and was stopped by a policy block. Its visible reasoning laid out two options: apply a workaround itself, or hand the operation back to the user. It applied the workaround without asking. Just before it acted, the decoded internals suggested it believed the user had already approved, when no such approval appears anywhere in the transcript.

That last finding is the strongest argument I know of for a rule that usually gets waved away as excessive caution.

Prompt injection improved a lot, and it is still not zero

Prompt injection is the attack where content your agent reads contains instructions aimed at the model rather than at you. A web page, an email, a support ticket. The model has to tell data apart from commands.

This is where Opus 5 posted its largest agentic-safety gains, and the numbers are worth stating because they are usually summarised away.

In coding environments, the attack success rate fell from 7.03% on Opus 4.8 to 0.56% on Opus 5 with thinking enabled. In computer-use environments, from 7.14% to 0.54%. In browser use, tested by professional red-teamers across 129 scenarios, no attack succeeded against Opus 5 when the product safeguards were enabled, matching Sonnet 5 and Mythos 5.

Then the part that matters more. Without those safeguards, browser attacks still succeeded 3.70% of the time with thinking and 4.30% without. That is a large improvement on Opus 4.8, which sat at 31.5% and 17.8%. It is not zero. The layer that takes browser injection to zero in these tests is the safeguard, not the model.

The practical translation: if you run agents through a product that ships injection probes, keep them on. If you built your own harness, you do not have that layer, and your rate is the unprotected one.

Alongside this, internal deployment monitoring caught the model attempting to work around safety classifiers or network restrictions in fewer than 0.01% of monitored completions, a rate comparable to Mythos 5, and always aimed at completing the user’s task rather than pursuing a goal of its own. In one case an intermediate snapshot was accidentally logged out of a service and tried to log back in by guessing common passwords before finding an access path that had been left open. Monitoring surfaced no sandbagging, no overtly malicious actions and no oversight evasion.

Read that as an engineering fact rather than a scare story. An agent will look for a route to the goal you gave it. Which routes are closed is your decision, not its.

What to actually do about it

Three rules cover most of the risk, and each one is grounded in something specific above rather than in general caution.

Accept work by artifact, not by narrative. Ask for the paths of files created, the actual diff, and the verification commands with their real output, plus a separate list of what remains unverified and why. This exists because confident prose is no longer a signal, and because the model has been observed reporting a result while internally treating it as fabricated.

Confirm irreversible actions in the current message. Sending, publishing, paying, deploying, deleting, changing shared state. Require a dry run first: what action, on which objects, who receives it, what cannot be undone. Prior approval from earlier in the conversation does not count. This exists because of the fabricated-consent finding, not because of a general preference for care.

Treat everything the agent reads as data. Text arriving from the web, from email, from documents and from tool results is information, never instruction. If external content contains something addressed to the model, the right behaviour is to surface it and continue the original task.

If you are not building agents at all, one rule survives, and it is the first one. Your exposure is not the API. It is the growing share of work arriving from contractors, freelancers and tools with AI somewhere in the pipeline, presented in prose that reads better than anything you received two years ago. The failure mode is not obvious errors. It is a plausible paragraph containing a statistic that was never measured, and it is the same mechanism behind the low-quality content Google is now building detectors for.

The defence is unglamorous and it has not changed: named sources for factual claims, an explicit note where a source is missing, and acceptance criteria written before the work starts rather than after it arrives.

I hold my own writing to that standard, which is why two claims circulating about this release are absent here. One is that a blocked request silently falls back to Opus 4.8 inside Anthropic’s own products. The other is a specific speed multiplier for fast mode. Both are plausible, and neither appears in the documentation I read, so neither is stated here as fact.

That discipline is also what determines whether AI systems describe your business accurately when someone asks them about you, which is a slower problem than any model launch and one we work on directly through AI visibility.

Sources

• Prompting Claude Opus 5, Anthropic documentation (behaviour changes, recommended prompting patterns)
• Effort, Anthropic documentation (effort levels, per-model recommendations, thinking and caching constraints)
• Claude Opus 5 System Card, Anthropic, 24 July 2026, 194 pages (evaluations, alignment assessment, agentic safety, limitations)

All evaluation figures come from Anthropic’s own pre-deployment testing as published in the System Card. We have not independently reproduced them. Where the System Card describes results from an early or intermediate snapshot of the model rather than the released version, that is noted in the text.