Cleo now runs on Claude Opus 4.8 — and we ran the eval to prove it matters.

Why a model upgrade matters for compliance specifically

A legal-drafting assistant can be judged on tone and structure. A compliance engine is judged on a binary that carries money and liability: can this product be sold in this market, and on the authority of which regulation? A fabricated regulation number, or a real one applied to the wrong product category, is not a stylistic blemish — it is a wrong answer that a brand might act on.

This is the same risk we wrote about in our piece on AI fabricating regulations. Opus 4.8 is positioned by Anthropic as their more honest flagship, and it posts the highest score recorded on their Legal Agent Benchmark — the first model to break 10% on the strict all-pass standard. The question for us was narrower: does that translate into fewer wrong compliance answers, on our kind of task?

The eval: 5 products, two models, every citation verified

We picked five product-and-market cases across the categories Cleo covers — cosmetics, toys, electronics, food supplements, children’s textiles — chosen because the right answer hinges on precise identifiers (a regulation number, an EN standard) that a weaker model tends to invent or misapply. We sent each model the identical prompt, closed-book (no retrieval), asking for a verdict and the applicable regulations. Then we verified every single cited identifier against the official source: Légifrance, EUR-Lex, the CEN standards catalogue.

Opus 4.8 got every verdict right, returned a clean structured answer every time, and cited regulations with roughly a third of the error rate — while using fewer tokens. On the hardest case (a mains Bluetooth speaker, the densest regulatory stack), Opus 4.7 ran long and was cut off before it returned a usable verdict; Opus 4.8 returned a complete, correct answer well within budget.

Two errors 4.7 made that 4.8 did not

The interesting failures were not invented numbers — they were real regulations applied to the wrong product. This is exactly the kind of error that survives a quick human glance, because the citation looks authoritative.

What this changes inside Cleo

Fewer misapplied regulations surfaced to a user is the headline, but two operational gains matter just as much. Opus 4.8 is the strongest agentic model Anthropic has tested (84% on Online-Mind2Web), with more efficient tool-calling — which is exactly what MARIA does when it traverses Legal Atlas across 177 jurisdictions: more reliable multi-step retrieval, fewer wasted steps. And note: this eval was closed-book, model alone. Inside Cleo the model is grounded on Legal Atlas, which closes most of the remaining citation gap.

The honest limits of this test

Five cases is illustrative, not a statistically powered benchmark — read it as a directional signal, not a leaderboard. It was a single run per case, closed-book, so it measures the model’s parametric knowledge rather than the full grounded system. And to be fair to 4.7: on the French supplement case it was actually more complete than 4.8 (it included the arrêté du 26 septembre 2016 that 4.8 omitted). The pattern across the set was still clear — 4.8 was more accurate, more precise on citations, and more efficient — which is why it is now the default behind MARIA.

Methodology and Opus 4.8 figures: Anthropic (Introducing Claude Opus 4.8). Citations verified against Légifrance, EUR-Lex and the CEN standards catalogue.