Polly Brings AI Debugging Into LangSmith & Notion Runs Untrusted Code at Scale

LangChain - Polly: AI-Assisted Debugging for Agent Traces in LangSmith#

Source: https://blog.langchain.com/polly-langsmith-ga/ Credibility: High (first-party product launch with clear description of capabilities)

What happened: LangChain launched Polly as generally available across all LangSmith plans—an AI assistant built directly into the LangSmith interface that reads and interprets agent execution traces. The product release solves a specific pain point that's emerged as agents scale: traces that run hundreds of steps deep with prompts spanning thousands of lines are practically unreadable by humans. Polly can scan a full trace, surface what caused a failure, and explain the context buried in the middle of a long execution.

Key capabilities:

• Trace comprehension at scale: Polly reads traces that are hundreds of steps deep—something humans can't efficiently scan manually
• Failure diagnosis: Identifies the specific step, tool call, or reasoning path that caused an unexpected output or error
• Contextual explanation: Explains why something went wrong, not just where in the trace it happened
• Generally available: Not limited to preview users—available across LangSmith plans now
• Embedded in LangSmith UI: Not a separate tool; integrated into the existing observability workflow

The product gap this closes:

The Feb 24 LangChain post documented the principle that "you don't know what your agent will do until it's in production"—requiring trace-based monitoring of behavior, not just outcomes. Polly addresses the practical bottleneck that principle creates: if traces are the diagnostic tool but traces are too long to read, engineers avoid using them. Polly makes trace-based debugging tractable.

The architectural pattern:

Polly is an AI wrapper around observability data. The pattern—use an LLM to interpret your LLM's execution logs—will likely become a standard layer in agent infrastructure. The alternative (engineers reading raw traces manually) doesn't scale. This is the same logic that drove log analysis tools in traditional software operations: the volume of signals exceeds human review capacity.

Why it matters for PMs:

Polly is the first product signal that AI-on-AI debugging (an AI explaining what another AI did wrong) is a shipped, production feature rather than a concept. For PMs building agent products, this changes the support and quality-control calculus: instead of engineers spending hours tracing failures, Polly compresses that to minutes. The PM implication is velocity—faster debugging cycles mean faster iteration on agent quality. For PMs evaluating LangSmith as infrastructure, Polly is a meaningful differentiator: debugging is the most painful part of agent development, and an embedded assistant addresses it directly.

Critical questions:

• How accurate is Polly's failure diagnosis? If it misidentifies the root cause 20% of the time, does it save time or create false confidence?
• Does Polly's trace interpretation add meaningful latency to the debugging workflow, or is it near-instant?
• Can Polly handle traces from agents built outside LangChain (custom LangGraph workflows, non-standard tool implementations)?
• What's the token cost of feeding a 500-step trace into Polly's context—does this make debugging expensive at scale?

Action you could take today:

If your team uses LangSmith and has been avoiding trace review because traces are too long to read, open the last 3 agent failures and let Polly interpret them. Compare Polly's diagnosis to what your engineer believes caused the failure. The discrepancy (or agreement) tells you whether to trust Polly for production debugging or treat it as a starting point that still needs human validation.

Vercel - How Notion Workers Run Untrusted Code at Scale with Vercel Sandbox#

Source: https://vercel.com/blog/notion-workers-vercel-sandbox Credibility: High (production case study from a recognizable customer, first-party Vercel blog)

What happened: Vercel published a case study documenting how Notion uses Vercel Sandbox to run untrusted code at scale inside Notion's product. Notion Workers—the feature that allows users to write and run custom JavaScript inside Notion—required secure, isolated execution for user-generated code. Notion's team chose Vercel Sandbox as the infrastructure layer to handle that isolation, rather than building their own.

Key technical patterns from the case study:

• The use case: Notion Workers lets users write JavaScript that runs inside their Notion workspace (automations, data transformations, custom integrations). Every script is user-generated, which means untrusted by definition.
• The scaling challenge: Executing untrusted code requires isolation—each execution must be sandboxed so a malicious or buggy script can't affect other users, access Notion's infrastructure, or leak data
• What Vercel Sandbox provides: Isolated V8 execution environments spun up on demand, with network and filesystem restrictions, that return results to the calling service
• Why buy vs. build: Building production-grade sandboxed execution infrastructure is a significant engineering investment—cold start performance, memory limits, escape prevention, concurrent execution at scale. Notion delegated this to Vercel Sandbox rather than building it internally

The buy-vs-build signal:

This case study directly addresses the open question about build vs. buy patterns for AI capabilities. Notion—a sophisticated engineering team—chose to buy sandbox infrastructure rather than build it. The implication: secure code execution is a genuine infrastructure problem that justifies delegation even for large engineering teams. The boundary for "worth buying" is not just about capability but about operational complexity: Notion's engineering team can focus on what Notion Workers does for users rather than how untrusted code is safely isolated.

The broader pattern:

Vercel Sandbox (launched Jan 31) + Notion Workers case study + LangSmith Sandboxes (March 17, covered in Mar 7 update) shows sandboxed execution converging as standard infrastructure for AI products. As agents and AI features that execute code become more common, the need for managed sandbox infrastructure scales proportionally. This is the same trajectory as managed databases or authentication: infrastructure that starts as DIY becomes a managed service as the problem scales.

Why it matters for PMs:

For PMs building products where users (or agents) execute code—automations, data transformations, custom logic, AI-generated scripts—this case study provides a concrete reference architecture. The decision tree: is your user-generated code execution use case small enough to handle with simple restrictions, or complex enough to need managed sandbox infrastructure? Notion's choice signals that even at enterprise scale, the answer is to buy rather than build sandbox execution. For PMs evaluating Vercel infrastructure, this is a proof point for Sandbox beyond demo scenarios.

Critical questions:

• What's the latency profile for Vercel Sandbox cold starts—are they acceptable for interactive Notion Workers executions, or does Notion only use it for background tasks?
• What's the cost structure—is Vercel Sandbox priced per execution, per compute-second, or flat rate?
• How does Notion handle debugging when a Worker fails inside a sandbox—do error messages surface clearly, or are they opaque?
• Are there execution time limits in Vercel Sandbox that constrain what Notion Workers can do (no long-running scripts, limited memory)?
• What was the alternative Notion considered—AWS Lambda, Cloudflare Workers, or building custom V8 isolates?

Action you could take today:

If you're building a product where users or agents execute custom code (automations, data transforms, AI-generated scripts), audit your current isolation model: is every execution sandboxed, or are you trusting user code more than you should? If isolation is missing or DIY, the Notion case study is the reference for evaluating Vercel Sandbox as managed infrastructure rather than building your own.

Teresa Torres - Conversations with Claude: Can You Conduct a Content Audit?#

Source: https://www.producttalk.org/conversations-with-claude-content-audit/ Credibility: High (first-party practitioner case study from recognized PM thought leader)

What happened: Teresa published a case study documenting her experiment using Claude to conduct a content audit of her producttalk.org blog—testing whether AI could systematically analyze 350+ posts for themes, gaps, quality patterns, and strategic recommendations. The post documents where Claude succeeded, where it failed, and what the workflow revealed about using AI for analytical product tasks.

Key patterns from the experiment:

What Claude did well:

• Systematic categorization of large content volumes (analyzing hundreds of posts for themes)
• Pattern identification across content that would take days to read manually
• Surfacing content gaps—topics discussed by others but missing from Teresa's archive
• Generating structured output (spreadsheets, categorization schemes) that could be reviewed and refined

Where Claude struggled:

• Quality judgment that requires domain expertise—distinguishing a "good" post from a "thorough" post requires understanding the audience, not just reading the text
• Detecting subtlety in argument quality—Claude could summarize posts but couldn't reliably evaluate whether an argument was well-reasoned versus superficially plausible
• Providing strategic recommendations that account for audience evolution—Claude didn't have context for how Teresa's audience had changed over time

The workflow that worked: Teresa iterated with Claude across multiple conversations rather than expecting one prompt to produce a complete audit. The pattern: Claude generates a first pass, Teresa evaluates and corrects, Claude refines based on corrections. This matches the memory/learning loop pattern from LangChain's Feb 23 architecture post—iterative correction as the improvement mechanism.

The PM craft insight:

Teresa frames the content audit as analogous to a product audit: systematically reviewing your existing work to understand what's there, what's missing, and what should change. The AI-assisted version compresses the time to pattern recognition while maintaining human judgment for strategic decisions. The takeaway: AI accelerates the data processing layer of audits while humans remain essential for the judgment layer.

Why it matters for PMs:

This provides a concrete, first-person playbook for using AI in analytical PM tasks. Content audits, feature audits, user feedback categorization, competitive analysis—all follow the same pattern: large volumes of text that benefit from AI-powered systematic analysis, followed by human judgment on what the patterns mean. Teresa's documentation of where Claude succeeded and failed is more useful than a general claim that "AI helps with research." For PMs trying to use AI for discovery and strategy work, this is a worked example.

Critical questions:

• How does this workflow scale for PM tasks with higher stakes than content strategy—user research synthesis, bug triage, customer feedback categorization where misclassification has product consequences?
• What's the right human review rate—does Teresa review every Claude categorization, or only edge cases?
• Does using Claude for content audits create a selection bias toward the things Claude can easily categorize versus what actually matters strategically?
• At what point does iterating with Claude across multiple conversations become more expensive in time than doing the analysis manually?

Action you could take today:

Pick one PM artifact you've been meaning to audit but haven't because the volume is daunting—backlog of user feedback, support ticket themes, competitive feature comparisons. Run a version of Teresa's workflow: give Claude a sample (20-30 items), ask for categorization with rationale, evaluate the categorization against your own judgment, correct it, then scale to the full set. The goal isn't to trust Claude's output—it's to test whether Claude's first-pass categorization accelerates your analysis or creates noise.