How PMs Are Actually Using AI Coding Tools & Agent Framework Necessity

Lenny Rachitsky - How PMs Are Actually Using AI Coding Tools: Community Survey Results#

Source: https://www.lennysnewsletter.com/p/community-wisdom-how-pms-are-using Credibility: High (community survey with concrete PM usage patterns)

What happened: Lenny published survey findings on how PMs actually use AI coding tools—revealing a gap between hype and practice. The headline: most PMs use AI for research and prototypes, not production features.

Key usage patterns from survey respondents:

What PMs actually do with AI coding tools:

• Research and experimentation: Build throwaway scripts to test hypotheses or analyze data
• Quick prototypes: Generate UI mockups or basic functionality to validate concepts
• Data analysis: Write Python scripts for cleaning data, running queries, or visualizing metrics
• Documentation: Generate API specs, README files, or technical documentation
• Internal tools: Build Slack bots, admin panels, or workflow automation

What PMs don't do (yet):

• Ship production features: Most survey respondents hand AI-generated code to engineers for review/rewrite
• Complex business logic: Avoid using AI for critical paths (payments, auth, core features)
• Database migrations: Reluctance to let AI modify production schemas
• Performance-critical code: Concerned about AI-generated code being inefficient

The confidence gap: Survey respondents report using AI tools frequently but lack confidence in output quality. Pattern: "I use it daily for prototypes but wouldn't ship it to customers without engineer review."

Why the gap exists:

• Quality uncertainty: Can't reliably assess when AI code is good enough for production
• Organizational barriers: Engineering teams prefer their own code or require review of AI-generated code
• Risk aversion: Stakes are too high for customer-facing features
• Skill limits: PMs can describe features but can't debug when AI code breaks

The actual productivity gain: AI tools accelerate exploration and validation, not deployment. The value: PMs test more ideas faster, not ship more features solo. This validates the "research and prototype" use case over "replace engineers."

Why it matters for PMs: This grounds the vibe coding narrative (Feb 8, Feb 12) in reality. Despite tools like Claude Code and v0, most PMs use AI for experimentation—not autonomous feature shipping. For PM leaders evaluating AI tool adoption, the question is: do you want faster prototyping or actual feature deployment? Current practice optimizes for the former.

Critical questions:

• What would it take for PMs to trust AI-generated code for production? Better testing? Better AI? Different product architecture?
• Is the prototype-to-production handoff actually slower than PMs building prototypes in Figma or slides? Or does code accelerate validation?
• Which PM workflows benefit most from AI coding—those requiring data analysis, UI prototypes, or process automation?

Action you could take today: If you manage PMs, survey your team: how are they actually using AI coding tools? Are they prototyping, researching, or shipping? If mostly prototyping, optimize tooling and training for that use case rather than expecting production-ready output.

LangChain - Why Agent Frameworks Still Matter Despite Better Models#

Source: https://blog.langchain.com/on-agent-frameworks-and-agent-observability/ Credibility: High (detailed technical argument with architecture analysis)

What happened: LangChain published a defense of agent frameworks—arguing that despite model reasoning improvements, frameworks remain essential for production systems. This was already covered in the Feb 14 update.

Deduplication note: This item appeared in full detail in yesterday's update (Feb 14). The core argument: models handle reasoning; frameworks handle orchestration, state, observability, and safety. Not repeating full analysis here.

Why it's relevant today: LangChain's framework defense directly addresses the question raised by Lenny's PM survey findings: if PMs struggle to ship AI-generated code to production, is it because they lack frameworks (LangChain's claim) or because AI output quality isn't trusted regardless of infrastructure?

The tension:

• LangChain argues frameworks are production-critical
• Lenny's survey shows PMs avoid production deployment even with frameworks available
• Gap suggests trust problem, not infrastructure problem

Action you could take today: If you're building agents but hesitant to deploy, diagnose whether your blocker is infrastructure (missing observability, state management) or trust (uncertain quality). If infrastructure, evaluate frameworks. If trust, focus on testing strategies and quality gates.