Home
May 7, 2026
View All

Anthropic's Dev Conference, Windsurf's Code Review, and Validating Agents

The Short Version#

Anthropic's developer conference dropped major Claude updates while Dario Amodei's 80x growth number shows what software-first AI adoption actually looks like at scale — and two complementary pieces on agent validation (from GitHub and Windsurf) land the same week, giving PMs a real framework for how to think about correctness when agents are doing the work.

Anthropic — Code with Claude 2026 Developer Conference#

Source: https://www.lennysnewsletter.com/p/code-with-claude-the-5-biggest-updates / https://simonwillison.net/2026/May/6/code-w-claude-2026/#atom-everything Credibility: High (conference keynote + live blog by Simon Willison, Lenny's video recap)

What happened: Anthropic held its Code with Claude developer conference in San Francisco today. Dario Amodei opened with a number that stopped people: Anthropic planned for 10x revenue and usage growth in Q1, and ended up at 80x. His framing — "software engineers are the ones who are fastest to adopt new technology, and this is a foreshadowing of how things will work across the economy" — is less a brag than a thesis about where enterprise AI adoption goes next. Simon Willison live-blogged the event; Lenny put out a recap of the five biggest updates from the Claude platform announcements.

Key capabilities announced:

  • Claude Code expanded significantly (agent-level coding, not just autocomplete)
  • New developer-facing features across the Claude platform (exact details in Lenny's recap video and Simon's live blog)
  • Dario's Jevons Paradox framing at the May 5 financial services event: "If you automate 90% of the job, then everyone does the 10% of the job" — positioning AI as demand amplifier, not job destroyer

Why it matters for PMs: The 80x growth number is the headline but the framing is the signal. When a foundation model company talks about engineers being "first movers" and calls it a "foreshadowing," they're telling you where the adoption playbook goes next — into every other function. If you're building B2B AI products, your customers' software teams are already ahead of their own roadmap because of tools like this. The Jevons Paradox quote also matters: if you're designing features around "AI saves time," you may be solving the wrong problem. The more interesting design question is what happens to the 10% that humans still own.

Critical questions:

  • What specifically shipped at the conference vs. what was announced as roadmap? The excerpt is thin on product details — worth reading the full Lenny and Simon writeups for the actual feature list.
  • 80x growth sounds extraordinary. What's the baseline? A 10x plan that became 80x from a small number is different from 80x at scale.
  • The Jevons Paradox framing is intellectually interesting but does it hold for real enterprise workflows? If you automate 90% of a job, do people actually expand into new 10% tasks, or does headcount just shrink?
  • How do the new Claude Code features compare to what Cursor and Windsurf shipped this week?

Action you could take today: Read Simon Willison's live blog (https://simonwillison.net/2026/May/6/code-w-claude-2026/#atom-everything) alongside the Lenny recap to separate the shipped product changes from the conference narrative — then flag which Claude Code capabilities are already available vs. coming soon.

GitHub Blog — Validating Agentic Behavior When "Correct" Isn't Deterministic#

Source: https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/ Credibility: High (first-party GitHub engineering blog, published May 6)

What happened: GitHub's AI/ML team published a post directly addressing one of the hardest unsolved problems in shipping agents to production: how do you define "correct" when agent outputs are non-deterministic? The piece explores validation frameworks for agentic systems where there's no single right answer — the agent could take multiple valid paths to complete a task.

Key patterns from the piece:

  • Traditional software testing (input → expected output) breaks for agents because the same prompt can produce different but equally valid sequences of actions
  • The post advocates for behavioral validation — testing that agents satisfy constraints rather than exact outputs (e.g., "the task was completed without deleting any user data" rather than "the agent took these exact steps")
  • Introduces the concept of invariant checking: properties that must always hold regardless of which valid path the agent takes
  • Suggests trace-level evaluation — logging the full action sequence and evaluating each step, not just the final result
  • Proposes test harnesses that run agents multiple times and check for consistency across runs, even when outputs differ

Why it matters for PMs: This is the gap between demos and production. Every agentic feature looks great in a demo because you control the prompt. The failure mode is when users prompt slightly differently and the agent takes a path your QA didn't anticipate. The behavioral validation framing is the practical mental model: stop asking "did the agent produce the right output?" and start asking "did the agent stay within the right constraints?" This also has implications for how you write acceptance criteria for agentic features — outcome-based, constraint-defined, not step-by-step.

Critical questions:

  • How do you set constraints without accidentally over-specifying the solution space and killing the agent's flexibility?
  • What does the tooling look like for trace-level evaluation at scale? This approach gets expensive quickly if agents are taking 20+ steps per task.
  • Does GitHub use this framework internally for Copilot's agentic features, or is this theoretical?
  • How do you handle invariant violations in production — rollback, human escalation, or silent failure?

Action you could take today: Take one agentic feature you're currently building or planning and rewrite its acceptance criteria using the behavioral/constraint model: instead of "agent does X," write "agent completes the task without doing Y or Z." See if that changes what you'd actually test.

Windsurf — Code Review Comes Into the Editor#

Source: https://windsurf.com/blog/devin-review-windsurf Credibility: High (first-party product blog, shipped feature, May 6)

What happened: Windsurf shipped two code review features — Devin Review and Quick Review — that bring verification directly into the editor rather than requiring a separate PR or external review step. The pitch: "verification in the same workspace where you write code." Devin Review appears to be a more comprehensive analysis; Quick Review is the fast-path option.

Key capabilities:

  • Code review happens inside Windsurf, not in a separate PR tool or code review platform
  • Two modes: Devin Review (comprehensive) and Quick Review (fast)
  • Designed to reduce context switching between writing code and getting it reviewed
  • Positioned as a complement to AI code generation — generate with Cascade, verify with Devin Review, in the same window

Why it matters for PMs: This is the workflow integration play, not a net-new capability. Code review has always existed; what Windsurf is doing is collapsing the "write → submit → wait → review → fix" loop into a tighter feedback cycle. If your team is evaluating AI coding tools, this is a concrete differentiator to test: does having review in the same tool actually reduce cycle time, or does it create a false sense of thoroughness? It also connects to the GitHub post above — if you're validating agentic behavior, having review tools closer to generation is exactly the kind of guardrail that matters.

Critical questions:

  • What's the actual review quality? AI-generated code reviewed by another AI pass in the same tool has a real echo-chamber risk — does Devin Review catch what Cascade misses, or does it rubber-stamp it?
  • How does this interact with existing PR workflows and team code review processes? Is it additive or does it encourage skipping human review?
  • Is Quick Review actually useful or is it a watered-down version that gives developers false confidence?
  • Windsurf is in an acquisition/transition period (OpenAI rumored acquirer) — how does product roadmap continuity look?

Action you could take today: If your team uses Windsurf, run the same piece of AI-generated code through both Quick Review and a manual human review and compare what each catches. That gap is your actual risk surface for AI coding at speed.

Quick Hits#

  • Lenny Rachitsky: Video recap of everything Anthropic launched at Code with Claude — five biggest updates explained. (2026-05-07): https://www.lennysnewsletter.com/p/code-with-claude-the-5-biggest-updates

  • Notion: Launched a Custom Agent Directory — a curated listing of available custom agents following the beta period. Pairs with their earlier "What we learned during the Custom Agents beta" post. (2026-05-06): https://www.notion.so/releases/2026-05-06

  • Cursor: Shipped Context Usage Breakdown — agents now show a breakdown of context consumption across rules, skills, MCPs, and subagents to help diagnose context issues. Small but directly actionable for teams managing complex Cursor setups. (2026-05-06): https://cursor.com/changelog/05-06-26

  • Dan Shipper: "It's fairly trivial to 0 a model on any benchmark by changing the frame — few understand this." Sharp reminder that benchmark performance is a framing artifact, not a fixed property of a model. Directly relevant to anyone using benchmarks to make model selection decisions. (2026-05-05): https://x.com/danshipper/status/2051698999619690814

  • Harrison Chase: "We need more benchmarks" in response to Harvey's benchmark work for legal AI. LangChain's founder pushing for domain-specific evaluation — the pattern that real-world AI evaluation needs task-specific harnesses, not general leaderboards. (2026-05-06): https://x.com/hwchase17/status/2052063837072065001

The Thread#

Validation is the new shipping problem. Three signals this week converge on the same gap: GitHub's post on non-deterministic agent correctness, Windsurf bringing review into the editor, and Harrison Chase calling for more domain-specific benchmarks. When generation gets cheap and fast, the constraint becomes trust — and trust requires verification infrastructure that doesn't exist yet at scale. The teams figuring out how to validate agentic behavior without slowing everything down are the ones who'll be able to ship agents into production with confidence.

Sit With This#

GitHub's post argues that behavioral validation — testing constraints rather than exact outputs — is the right framework for agentic systems. Instead of "did the agent do X?", ask "did the agent avoid Y and Z?"

For your product: Pick one agentic feature you're building or evaluating. What are the invariants — the things that must never happen, regardless of which path the agent takes? Can you actually test for those today, or are you only testing the happy path?