Building Agent Test Harnesses & Teresa's AI for Opportunity Trees

LangChain - Improving Deep Agents with Harness Engineering#

Source: https://blog.langchain.com/improving-deep-agents-with-harness-engineering/ Credibility: High (detailed technical breakdown with benchmark results)

What happened: LangChain documented how they improved their coding agent from Top 30 to Top 5 on Terminal Bench 2.0—without changing the underlying model. The entire improvement came from "harness engineering": optimizing how the agent interacts with its environment.

Key harness engineering patterns:

What a harness is: The infrastructure layer between the model and the task. For coding agents, the harness includes:

• How the agent receives instructions and context
• Which tools it can access (terminal, file editor, browser)
• How it receives feedback from code execution
• When and how it retries failed attempts

The two breakthrough optimizations:

1. Self-verification loops: Instead of executing code once and moving on, the agent now:

• Runs code
• Checks if output matches expected behavior
• If mismatch, analyzes the error
• Retries with correction
• Repeats until success or max attempts

Implementation detail: The verification step uses the same model but with a different prompt: "Does this output match what we wanted? If not, what's wrong?"

2. Tracing-informed debugging: When code fails, the agent reviews its own LangSmith traces to understand:

• Which tools it invoked
• What intermediate outputs it received
• Where reasoning diverged from correct path

The agent uses traces as debugging logs—something humans do naturally but agents typically skip.

Why this matters more than model improvements: Model quality matters, but Terminal Bench performance jumped 25+ positions through harness changes alone. The implication: agent capability is often harness-limited, not model-limited.

The architectural insight: Most teams focus on prompting or model selection. LangChain argues the harness—how you structure agent-environment interaction—matters more. Better harnesses unlock capability from existing models.

Why it matters for PMs: This reframes the "improve agent performance" problem. Instead of waiting for better models or writing better prompts, optimize the harness: add verification loops, expose better feedback, enable self-debugging. For PMs building agents, the question becomes: what constraints in your harness are limiting performance?

Critical questions:

• What's the latency overhead of self-verification loops? Does retrying 3-5 times per task make agents too slow?
• How do you prevent verification loops from creating infinite retries when the task is genuinely impossible?
• Does this pattern work for all agent types, or just coding agents with deterministic outputs?
• At what point does harness complexity create more problems than it solves?

Action you could take today: If you're building agents, audit your harness: does the agent retry failed actions? Does it verify outputs before moving on? Does it have access to its own traces for debugging? Implement one verification loop for your highest-failure task and measure whether success rate improves.

Teresa Torres - From Customer Interviews to Opportunity Solution Trees—In Minutes#

Source: https://www.producttalk.org/ai-opportunity-solution-trees/ Credibility: High (detailed workflow with concrete examples)

What happened: Teresa published a guide showing how AI can structure messy interview transcripts into opportunity solution trees—collapsing a multi-hour manual process into 5-10 minutes of AI-assisted work.

The workflow pattern:

Traditional approach (2-4 hours per interview):

1. Read transcript (30-60 minutes)
2. Highlight key insights (15-30 minutes)
3. Extract opportunities (30-60 minutes)
4. Cluster related opportunities (30-60 minutes)
5. Draft opportunity solution tree structure (30-60 minutes)

AI-assisted approach (5-10 minutes per interview):

1. Upload transcript to Claude or GPT
2. Prompt: "Extract opportunities from this interview"
3. Review AI-generated opportunities
4. Prompt: "Organize these into an opportunity solution tree"
5. Refine tree structure through conversation

Key quality improvements from AI:

• Consistency: AI applies the same opportunity extraction pattern across interviews (humans drift)
• Completeness: AI catches opportunities humans miss in long transcripts
• Speed: 5-10 minutes versus 2-4 hours enables processing more interviews

What AI still can't do:

• Judgment on priority: AI lists opportunities; humans decide which matter most
• Strategy alignment: AI doesn't know your product strategy; humans connect opportunities to goals
• Synthesis across interviews: AI processes one transcript at a time; humans identify cross-interview patterns

The human-AI collaboration model: AI handles extraction and initial structuring (high-volume, low-ambiguity work). Humans handle prioritization and strategic synthesis (low-volume, high-judgment work).

Why it matters for PMs: This continues the "automate execution, preserve judgment" pattern from previous updates. Teresa's workflow doesn't replace discovery—it accelerates the mechanical parts so PMs spend more time on strategic synthesis. For PMs doing continuous discovery, the question is: which parts of your process are high-volume but low-ambiguity?

Critical questions:

• What's the error rate on AI-extracted opportunities? How often does AI miss important insights or fabricate non-existent opportunities?
• Does AI bias toward explicitly stated opportunities versus implicit user needs?
• At what interview length or complexity does AI extraction quality degrade?
• How do you validate AI-generated trees without re-reading the full transcript?

Action you could take today: Take one customer interview transcript. Run Teresa's workflow: extract opportunities with AI, generate tree structure, then manually review. Compare time spent (5-10 minutes) versus manual extraction (2-4 hours). Calibrate whether quality trade-offs justify speed gains.

Vercel - How Stably Ships AI Testing Agents in Hours, Not Weeks#

Source: https://vercel.com/blog/How-Stably-ships-AI-testing-agents-in-hours-not-weeks Credibility: High (detailed case study with customer implementation)

What happened: Vercel published a case study on Stably—showing how they use Vercel Sandboxes to ship AI testing agents that automate end-to-end test creation in hours rather than weeks of manual QA work.

Key implementation patterns:

The testing agent workflow:

1. Agent receives feature spec or product change
2. Generates test scenarios (happy path, edge cases, error states)
3. Writes test code (Playwright, Selenium, or custom frameworks)
4. Executes tests in sandbox environments
5. Reports results and failure diagnostics

Why sandboxes matter here: Testing agents need isolated environments to run tests without affecting production or development systems. Vercel Sandboxes provide:

• Instant provisioning: Spin up test environment in seconds
• Clean state: Each test runs in fresh environment
• Resource isolation: Tests don't interfere with each other
• Cost optimization: Only pay for active test execution time

The speed claim breakdown:

• Traditional QA: 2-3 weeks to write comprehensive test suite for new feature
• Stably's agents: 2-4 hours to generate and execute test suite
• Speed multiplier: 60-100x faster

What the agent doesn't replace:

• Test strategy: Humans define what to test and coverage goals
• Failure analysis: Humans investigate root causes of test failures
• Quality judgment: Humans decide when coverage is sufficient

Why it matters for PMs: This validates agent infrastructure (sandboxes, orchestration, observability) as production-critical for agent products. Stably's agents work because the infrastructure supports rapid test execution. For PMs building agents, the question is: what infrastructure constraints limit your agent's usefulness?

Critical questions:

• What's the false positive rate on agent-generated tests? Do tests catch real bugs or just increase noise?
• How do teams maintain agent-generated tests over time as features evolve?
• At what scale does agent-generated test overhead exceed human-written test benefits?
• Does this work for all test types (unit, integration, E2E) or only specific scenarios?

Action you could take today: If you're building features that require testing, prototype agent-generated tests for one feature. Measure: time to generate, test quality, and false positive rate. This calibrates whether agents actually accelerate testing or just shift work to test maintenance.