Building AI Product Sense & LangSmith on Google Cloud

Lenny Rachitsky - Building AI Product Sense, Part 2: Pattern Recognition in Practice#

Source: https://www.lennysnewsletter.com/p/building-ai-product-sense-part-2 Credibility: High (detailed framework with specific examples and exercises)

What happened: Lenny published Part 2 of his AI product sense series—moving from principles (Feb 3) to practice. The core framework: AI product sense develops through deliberate pattern analysis across domains, not passive consumption.

Key skill-building exercises:

1. Comparative product analysis: Pick two AI products solving similar problems. Document:

• What prompts trigger similar outputs versus different ones?
• Where do quality differences emerge (latency, accuracy, context handling)?
• Which architectural choices create those differences?

Example exercise: Compare ChatGPT and Perplexity for research tasks. Perplexity spawns subagents (Jan 28 pattern); ChatGPT uses single-context deep reasoning. Result: Perplexity handles broader research; ChatGPT handles deeper analysis.

2. Failure mode cataloging: Maintain a running list of how AI products fail:

• Context window filling → quality degradation (Teresa's context rot, Feb 4)
• Ambiguous instructions → agent confusion (Lenny's Clawdbot, Jan 28)
• Tool selection errors → wrong approach chosen
• Recovery patterns: which products handle failures gracefully?

The practice: catalog failures you encounter daily. After 30 days, patterns emerge—common failure modes across products reveal architectural constraints.

3. Constraint-to-capability mapping: For each technical constraint (latency, cost, context limits), identify which product decisions follow:

• High latency → progressive disclosure (stream partial results)
• High cost → routing (use cheap models for simple, expensive for complex)
• Context limits → subagents or memory systems (Jan 29 patterns)

4. User feedback synthesis: Read user reviews of AI products critically. Ask:

• Which complaints reveal product decisions versus model limitations?
• Which praise identifies differentiation versus expected capabilities?
• What unmet needs appear consistently across products?

The meta-skill: connecting dots across domains: Strong AI product sense means seeing a pattern in one product and recognizing how it applies elsewhere. Example: LangChain's subagent pattern (Jan 28) applies to Perplexity's Deep Research (Feb 5) and Earmark's meeting automation (Feb 5).

Why it matters for PMs: This operationalizes the "build product sense" advice from Feb 3. Instead of vague "use products more," Lenny provides specific exercises. The comparative analysis exercise alone—systematically testing two products side-by-side—builds intuition faster than years of casual use. For PMs wanting to improve AI judgment, these are actionable practices with measurable skill gains.

Critical questions:

• How long does each exercise take? Are these 15-minute daily practices or 2-hour weekly deep dives?
• Which exercises accelerate junior PM learning most versus which help senior PMs refine existing intuition?
• Can you build product sense through exercises alone, or do you need to ship products to develop judgment?

Action you could take today: Pick one exercise from the list above and commit to practicing it daily for one week. Start with comparative product analysis (easiest to schedule): test two AI products on the same task every day, document quality differences, and identify which architectural choices create those differences. After 7 days, review your notes for patterns.

LangChain - LangSmith Now Available in Google Cloud Marketplace#

Source: https://blog.langchain.com/langsmith-is-now-available-in-google-cloud-marketplace/ Credibility: High (first-party marketplace launch announcement)

What happened: LangSmith launched in Google Cloud Marketplace—enabling enterprise customers to procure agent observability infrastructure through existing Google Cloud accounts. This follows the January patterns (Deep Agents, context management) by making LangSmith accessible to enterprise procurement workflows.

Key enterprise integration details:

Billing consolidation:

• LangSmith usage appears on Google Cloud invoices
• No separate vendor relationship or contract negotiation
• Committed use discounts from Google Cloud apply to LangSmith

Procurement simplification:

• Pre-approved in many enterprise cloud budgets
• Bypasses multi-month vendor evaluation processes
• Security and compliance already vetted by Google Cloud partnership

What this means for LangSmith adoption: Marketplace listing removes procurement friction—the primary blocker for enterprise AI tool adoption. For PMs at companies with Google Cloud commitments, this makes LangSmith an easier sell internally than tools requiring separate contracts.

The broader pattern—agent infrastructure becoming enterprise SaaS: LangSmith joining Google Cloud Marketplace validates agent observability as enterprise infrastructure, not experimental tooling. Compare to Vercel Sandboxes reaching GA (Jan 31)—the pattern is infrastructure providers (LangChain, Vercel) making agent components production-ready and enterprise-friendly.

Why it matters for PMs: This signals agent development moving from "build your own observability" to "buy enterprise-grade tooling." For PMs building agents, the question shifts from "should we instrument our agents?" to "which observability platform integrates best with our cloud provider?" The marketplace launch makes LangSmith the default answer for Google Cloud shops.

Critical questions:

• Pricing through marketplace versus direct—is there a markup or discount?
• Does marketplace listing include support tiers, or is support separate?
• AWS and Azure equivalents—when do similar marketplace listings appear?
• What percentage of LangSmith's enterprise customers were blocked by procurement before this?

Action you could take today: If you're building agents on Google Cloud and struggling with observability, evaluate LangSmith through the marketplace. The procurement path is now simplified—no vendor evaluation required if your company already uses Google Cloud Marketplace.

Vercel - How We Built AEO Tracking for Coding Agents#

Source: https://vercel.com/blog/how-we-built-aeo-tracking-for-coding-agents Credibility: High (detailed technical architecture with implementation specifics)

What happened: Vercel published technical details on Agent Execution Observability (AEO)—their system for tracking coding agent behavior across model invocations, tool use, and code execution.

Key architectural patterns:

Span-based tracing:

• Each agent action (prompt model, run tool, execute code) creates a span
• Spans nest hierarchically to show task decomposition
• Timestamps and latencies measured per span
• Errors and retries captured automatically

Tool usage tracking:

• Which tools agents invoke (bash, file editing, API calls)
• Tool invocation frequency and success rates
• Tool combinations—which tools are used together in workflows
• Error patterns—where do tools fail most often?

Model behavior analysis:

• Token usage per model invocation
• Reasoning steps before tool selection
• Context window utilization over time
• Model switching patterns (when does agent change models mid-task?)

Code execution monitoring:

• Sandbox lifecycle (creation, execution, cleanup)
• Resource usage (CPU, memory, disk)
• File operations (reads, writes, deletions)
• Network calls from sandboxes

Why AEO beats traditional metrics: Traditional monitoring (latency, errors, uptime) doesn't reveal agent reasoning. AEO exposes the decision path: why did the agent choose this tool? What information led to this code execution? Where did reasoning break down?

Why it matters for PMs: This continues the observability pattern from Jan 20 (LangSmith clustering insights) and Jan 29 (context management for deep agents). Vercel is documenting how to observe agent behavior at production scale. For PMs building coding agents, AEO provides the reference architecture for debugging and improving agent performance.

Critical questions:

• Storage costs—how much data does span-based tracing generate at scale?
• Real-time analysis—can PMs query AEO data during active sessions, or is it post-hoc?
• Privacy—when agents access user code/data, how do you observe without exfiltrating sensitive information?
• Actionability—what specific product decisions does AEO data inform versus just surfacing interesting patterns?

Action you could take today: If you're building coding agents, audit your current observability: do you track tool invocations, reasoning steps, and sandbox operations? If no, implement span-based tracing for one workflow. Start simple: log each agent action with timestamps and context. Analyze after one week to identify bottlenecks and failure patterns.