Evals as PRDs, Agent Observability, and Anthropic's Rough Week

Braintrust / Lenny's Newsletter — "Evals Are the Modern Version of a PRD"#

Source: https://www.lennysnewsletter.com/p/how-braintrust-uses-ai-agents-evals Credibility: High (Lenny's Newsletter interview with Ankur Goyal, CEO of Braintrust, a production eval platform)

What happened: Ankur Goyal joined Lenny's podcast to talk through how Braintrust approaches AI product development. The headline quote is the frame: "Evals are the modern version of a PRD." His argument is that the way you encode your product's taste — what good looks like, what failure looks like — is now through your evaluation suite, not a written requirements doc. Braintrust uses AI agents, evals, and CI together as a system: you write evals, run them in CI against every change, and use agents to generate test cases and catch regressions.

Key patterns:

• Evals as spec: The eval suite defines product behavior the way a PRD used to. If it's not in the eval, it's not in the product.
• Eval-driven iteration: Teams ship faster when they can run evals against every PR automatically, rather than waiting for human review of AI outputs.
• Agent-generated test cases: Braintrust uses agents to expand eval coverage — generating edge cases and adversarial inputs that humans wouldn't think to write.
• CI integration: Evals run in CI pipelines, not just in development. Regressions get caught before they hit production.

Why it matters for PMs: If you're shipping AI features, you need a way to define "done" that isn't vibes. The eval-as-PRD framing is practically useful: instead of writing acceptance criteria in prose, you write them as evaluations. This also changes what the PM's job looks like — you're not just writing requirements, you're co-authoring the eval suite with your engineers. That's a skills shift worth thinking about now.

Critical questions:

• How do you write evals for subjective outputs — tone, helpfulness, trust — where "good" is genuinely contested?
• Does eval-driven development create a Goodhart's Law problem? Teams optimize for the eval, not the actual product behavior.
• What does PM ownership of evals look like in practice? Is this a PM artifact or an engineering artifact?
• How much eval coverage is enough before you trust an agent to act autonomously in a production workflow?

Action you could take today: If your team is shipping an AI feature, write down three specific examples of output you'd call "good" and three you'd call "bad." That's the seed of an eval suite — and it's a conversation worth having with your engineers before you ship.

LangChain — 100x Cheaper Trace Judge + Align Evals + GTM Agent Case Study#

Source: https://www.langchain.com/blog/building-a-100x-cheaper-trace-judge-with-fireworks (+ https://www.langchain.com/blog/introducing-align-evals + https://www.langchain.com/blog/how-we-built-langchains-gtm-agent) Credibility: High (first-party LangChain blog posts, production case studies)

What happened: LangChain published a cluster of production-focused content today. The most technically interesting: they partnered with Fireworks to fine-tune an open model as a "trace judge" — a model that evaluates whether agent traces represent good or bad behavior. The result matches frontier model (GPT-4-class) performance at roughly 1/100th the cost. Separately, they shipped "Align Evals," a LangSmith feature that calibrates your evaluators against human preferences so your automated scores actually track what humans care about. And they published a case study on their own GTM agent — an internal agent that increased lead conversion by 250% while saving each sales rep 40 hours per month.

Key technical details:

• Trace Judge: Fine-tuned on production traces to categorize failures with confidence scores, causal chains, and fix recommendations. Works at 1% of frontier model cost.
• Align Evals: A LangSmith feature that takes human-labeled examples and calibrates automated evaluators to match human judgment. Closes the gap between "model scores it well" and "humans think it's good."
• GTM Agent: Built on LangGraph, integrated with LangSmith for monitoring. The 250% lead conversion improvement and 40 hours/rep/month savings are from LangChain's own internal deployment.

Why it matters for PMs: The trace judge story is a direct answer to a cost problem that kills eval adoption: running GPT-4 to evaluate every agent trace is expensive enough that teams skip it. If you can get the same accuracy at 1% of the cost, the calculus changes — evals become viable at production scale. The Align Evals feature tackles the other side of the problem: even cheap evals are useless if they don't track human judgment. And the GTM agent case study is one of the cleaner internal agent ROI stories I've seen — specific numbers, real workflow.

Critical questions:

• The trace judge performance claim ("matches frontier model") needs scrutiny — on what task distribution, with what failure modes? Domain-specific fine-tunes often generalize poorly.
• How does Align Evals handle preference drift? Human preferences on AI output quality are changing fast, which means your calibration data goes stale.
• The GTM agent 250% conversion improvement — what's the baseline? What's the counterfactual? Internal case studies from the vendor building the tool are inherently optimistic.
• At what scale does a fine-tuned trace judge make economic sense versus just using a cheaper API model directly?

Action you could take today: If you're running LangSmith, look at whether your evaluators are calibrated against real human feedback or against a frontier model score you're treating as ground truth. The Align Evals feature is worth a pilot if you have any labeled human evaluation data sitting around.

Anthropic — A Hard Week, On the Record#

Source: https://simonwillison.net/2026/Jun/15/axios-clashes-anthropics/#atom-everything + https://www.anthropic.com/news (statement on US government directive) Credibility: High (Simon Willison aggregating Axios reporting + Anthropic's own statement)

What happened: Two things converged this week that are worth understanding together. First, Anthropic published a formal statement on the US government directive to suspend all access to Claude Fable 5 and Mythos 5 — the export control action that's been in the news. Second, Axios published a piece (surfaced by Simon Willison) with sourced quotes suggesting internal personality clashes at Anthropic contributed to the model availability disruption. The Axios headline is "They screwed us" — attributed to a partner or customer affected by the suspension.

This sits on top of the already-covered Fable 5 walkback situation from earlier this week.

Why it matters for PMs: If you're building on Claude or evaluating Anthropic as a provider, this week is a data point in the vendor reliability conversation. Export control directives affecting model access are a new category of risk that wasn't on most PMs' radars six months ago — and it's now clearly real. The internal-conflict angle (if accurate) adds another layer: model availability disruptions aren't always technical. Sometimes they're organizational. Your API dependency on any single provider — especially one navigating regulatory pressure and internal tension simultaneously — is a risk that deserves a mitigation plan.

Critical questions:

• How does your product handle a sudden API provider outage or suspension? Do you have a fallback model strategy, or would this break the product?
• Is the US government export control directive a one-time event or a pattern? If AI model access becomes subject to geopolitical regulation, what does your provider diversification strategy look like?
• How much weight should you give to reported internal dysfunction at a provider when making dependency decisions?
• Anthropic's statement is carefully worded. What does the absence of a detailed explanation tell you about how they're managing this publicly?

Action you could take today: If Claude is in your critical path, spend 30 minutes sketching out a fallback plan — which other provider you'd route to, what the integration effort would be, and what user-facing behavior would change. Even a rough version of this is better than none.

AWS DevOps Agent — Custom SRE Agents and MCP/A2A Protocol Support#

Source: https://aws.amazon.com/about-aws/whats-new/2026/06/aws-devops-agent-custom-agents/ Credibility: High (official AWS What's New announcement)

What happened: AWS DevOps Agent shipped support for custom SRE agents, bring-your-own sub-agents, and headless access via MCP (Model Context Protocol) and A2A protocols. Teams can now automate recurring SRE workflows, connect their own agents to the DevOps Agent as sub-agents, and access DevOps Agent capabilities from external tools via standardized protocols. This is a meaningful expansion from a single-purpose agent to an agent orchestration platform.

Key capabilities:

• Custom SRE agents: Define domain-specific automation for recurring operational workflows (incident response, runbook execution, etc.)
• Bring-your-own sub-agents: Connect external agents to the DevOps Agent, making it a coordinator rather than a standalone tool
• MCP and A2A headless access: Lets other tools and agents call into DevOps Agent programmatically, without a UI
• This effectively makes DevOps Agent composable — it can receive work from other agents and delegate to specialized sub-agents

Why it matters for PMs: This is the incumbent cloud provider following the multi-agent composition pattern that's been emerging in frameworks like LangGraph and Mastra. When AWS ships something, it's usually because enough enterprise customers asked for it. The MCP/A2A protocol support is notable — it signals that these inter-agent communication standards are real enough for AWS to build against, which accelerates their adoption. For PMs thinking about agent architecture, the pattern of "orchestrator + specialized sub-agents + standard protocols" is becoming the enterprise default.

Critical questions:

• Custom SRE agents sound powerful, but who writes and maintains them? This pushes complexity toward the customer's engineering team.
• MCP and A2A are relatively young protocols — what's the lock-in story here? Are these truly open or AWS-flavored?
• How does bring-your-own sub-agent governance work? Can enterprise customers audit what external agents are doing when connected to AWS infrastructure?

Action you could take today: If your team is building or evaluating agent infrastructure on AWS, check whether your current architecture could benefit from treating DevOps Agent as an orchestration layer rather than a standalone tool. The headless MCP/A2A access is the piece worth experimenting with if you have other agents in the stack.