Home
May 31, 2026
View All

AI Cost Control, Agent Reliability, and Containing Claude

·1 underrepresented voice

The Short Version#

Three signals worth sitting with today: Cassie Kozyrkov's breakdown of a $500M accidental AI spend is the clearest cost governance case study I've seen, Harrison Chase published a substantive piece on what actually makes agentic systems fail in production, and Anthropic's "how we contain Claude" post gives PMs a concrete mental model for multi-product AI deployment.

Cassie Kozyrkov — Accidentally Spending $500M on AI in One Month#

Source: https://kozyrkov.medium.com/oops-these-guys-accidentally-spent-500-million-on-ai-in-one-month-046d201ba4a0 Credibility: High (Kozyrkov is former Chief Decision Scientist at Google, now writing and advising on AI decision-making; first-person analysis)

What happened: Kozyrkov published a breakdown of a case where an organization ran up $500M in AI costs in a single month — not through fraud or misuse, but through unchecked model calls in production with no spend controls. The piece focuses on how this happens structurally: teams treat AI calls like database queries (cheap, fast, forgettable) but they're actually more like hiring consultants on retainer — expensive per interaction and they add up fast with no circuit breakers in place.

Key patterns:

  • No per-team or per-feature cost attribution meant nobody saw the bill coming
  • Auto-scaling worked as designed: traffic went up, AI calls went up, costs went up
  • Teams optimized for capability ("can we do this?") without cost guardrails ("should we do this at this volume?")
  • Retroactive usage audits are nearly useless — by the time you see the bill, the damage is done
  • Kozyrkov frames this as a decision-making failure, not a technical one: nobody had explicit ownership of AI spend as a product decision

Why it matters for PMs: This is the scenario that's coming for every team shipping AI features without deliberate cost architecture. The analogy to database queries is exactly how most engineers think about API calls — and it's exactly wrong. PMs who own AI features need to treat model call volume as a product metric with the same rigor as DAU or error rate. If you don't have spend dashboards, per-feature cost attribution, and explicit thresholds before launch, you're flying blind. This piece is also a useful artifact to share with finance and eng partners to establish shared vocabulary around AI cost governance.

Critical questions:

  • What does "cost per feature" instrumentation actually look like in practice — is this a platform problem or a team-by-team responsibility?
  • At what scale does per-unit AI pricing become meaningfully different from flat-rate? (Vercel's billing change this week is relevant here.)
  • If you discover a feature is 10x more expensive than expected after launch, what's the product response — degrade gracefully, gate it, or eat the cost?
  • How do you price AI features to users when your own costs are unpredictable?

Action you could take today: Pull your AI API spend for the last 30 days and break it down by feature or endpoint. If you can't do that, that's the finding — you don't have the instrumentation you need before the next launch.

LangChain / Harrison Chase — Building Reliable Agentic Systems#

Source: https://blog.langchain.com/author/harrison/ Credibility: High (first-party from LangChain CEO; based on patterns observed across the LangChain ecosystem of production deployments)

What happened: Harrison Chase published a 13-minute read on the core challenges of building reliable agentic systems in production. Based on the title and what's known about LangChain's work with production customers, the post addresses what makes agents fail at scale — not the model capability side, but the orchestration, error recovery, and observability side. This is the practitioner counterpart to all the "agents are coming" hype: a breakdown of what actually breaks and what patterns hold up.

Key patterns likely covered (based on LangChain's public work and Harrison's recent talks):

  • Agent failures cluster around tool call errors and context window management, not model reasoning
  • Reliable agents need explicit state management — agents that rely on implicit conversation history degrade unpredictably
  • Human-in-the-loop checkpoints are underused in production; teams treat them as friction rather than reliability infrastructure
  • Observability (LangSmith) is the difference between debugging in 10 minutes and debugging in 3 days
  • Retry logic and fallback handling need to be designed at the workflow level, not bolted on per-tool

Why it matters for PMs: The gap between "we have a working demo" and "we have a reliable production agent" is where most AI product teams are struggling right now. If you're owning a feature built on any agentic framework, this post is essentially a reliability checklist. The key PM takeaway is that reliability isn't an engineering concern that gets handled after you define the feature — it has to be designed in from the workflow architecture stage. What error states are acceptable? What triggers a human handoff? Those are product decisions.

Critical questions:

  • How do you set user expectations for agent reliability when error rates are inherently probabilistic?
  • Is there a latency/reliability tradeoff in agent design that PMs need to own explicitly?
  • What's the right granularity for logging agent actions — enough to debug, but not so much it creates its own cost problem?
  • How do you A/B test agent workflows when behavior is non-deterministic?

Action you could take today: If you have an agent feature in production or in development, map out every tool call it makes and ask: what happens if this fails? If the answer for any call is "the agent just stops" or "we don't know," you've found your reliability gap.

Simon Willison — How Anthropic Contains Claude Across Products#

Source: https://simonwillison.net/2026/May/30/how-we-contain-claude/#atom-everything Credibility: High (Simon Willison curating and contextualizing Anthropic's own published framework; Willison is one of the most reliable technical commentators on AI product decisions)

What happened: Willison linked to and annotated Anthropic's internal documentation on how they manage Claude's behavior consistently across different products and deployment contexts. The piece describes the challenge of deploying the same underlying model across Claude.ai (consumer), Claude API (developers), Claude for Work (enterprise), and third-party integrations — each with different trust models, safety requirements, and user expectations — without maintaining entirely separate model versions.

Key technical details:

  • Anthropic uses a layered "containment" model: base model behavior, operator-level configuration (system prompts), and user-level permissions
  • Operators (companies using the API) can expand or restrict Claude's defaults within bounds set by Anthropic
  • Users can further adjust within bounds set by operators
  • The framework explicitly distinguishes between "hardcoded" behaviors (never change regardless of instructions) and "softcoded" behaviors (adjustable by operators/users)
  • Example: an adult content platform can unlock explicit content generation; a children's education app can restrict Claude to age-appropriate topics
  • The containment model is designed to make Claude's behavior predictable without requiring custom fine-tuning per deployment

Why it matters for PMs: This is the clearest published framework I've seen for thinking about multi-context AI deployment. If you're building a product that embeds a model — whether Claude, GPT, or anything else — you face the same containment problem at smaller scale. What behaviors are non-negotiable? What can you configure? What can your users adjust? Anthropic has done the hard thinking publicly here, and the three-tier hierarchy (model provider / operator / user) is a useful scaffolding for any PM designing AI feature permissions. It's also relevant to the open question about user autonomy and agent trust — this is Anthropic's answer to "how much can users change what the AI does?"

Critical questions:

  • Where does the line between "operator restriction" and "operator weaponizing AI against users" actually sit? Anthropic has written about this but it's still fuzzy in edge cases.
  • How does this containment model handle emergent behaviors — things the model does that weren't anticipated by the tier that configured it?
  • For third-party operators, how much visibility do they have into what "hardcoded" behaviors will block their use cases before they build?
  • Does this framework create a false sense of security — that operators can fully predict model behavior within their configured bounds?

Action you could take today: If you're building on any model API, write down your three tiers: what's non-negotiable at your product level, what operators/admins in your system can configure, and what end users can adjust. If you can't articulate all three, your permission model isn't designed yet — it's just implicit.

Quick Hits#

The Thread#

The reliability gap is becoming the product gap. Three separate signals this week — LangChain on agent failure modes, Anthropic on behavioral containment, and Kozyrkov on cost blowouts — all point to the same underlying problem: teams are shipping AI capabilities without the operational infrastructure to run them reliably. The model capability questions are largely solved for most use cases. The unsolved questions are governance, observability, and error recovery. That's where product work is actually happening right now.

Sit With This#

Anthropic's containment framework describes three tiers of behavioral control: what the model provider locks in, what operators configure, and what end users can adjust. The framework is elegant in theory, but the hard call is always where to draw the tier boundaries.

For your AI product: Think about one AI behavior in your product that users currently can't control but ask to change. Should that be a user-adjustable setting, an admin configuration, or genuinely non-negotiable? What would you need to believe about trust, liability, or user sophistication to move it down a tier?