Home
Aug 26, 2026
View All

Proving AI ROI and the App Oversaturation Problem

The Short Version#

Two threads worth pulling today: LangChain published a concrete framework for proving agentic AI ROI in financial services (finally, something you can bring to a CFO), and Pieter Levels is ringing an alarm bell that deserves more attention than it's getting — more apps are being built with AI than ever, but sales are going down.

LangChain — How to Prove Agentic AI ROI in Financial Services#

Source: https://www.langchain.com/blog/proving-the-roi-of-agentic-ai-in-financial-services Credibility: High (first-party case study from LangChain, focused on production implementations using LangSmith)

What happened: LangChain published a detailed breakdown of how to measure and communicate ROI for agentic AI in financial services — specifically using RFP automation and AML compliance as example use cases. The piece covers how to map AI agent activity to business KPIs, track costs at the task level, and build governance structures that make the numbers defensible to finance and legal stakeholders. It's framed around using LangSmith and Pay-i together, but the framework applies broadly.

Key patterns:

  • Tie agent outputs to business KPIs that already exist — don't invent new metrics that require explanation
  • Track cost at the task level (per RFP processed, per AML flag reviewed), not just total infrastructure spend
  • Build governance artifacts that show what the agent did, why it did it, and what the human did next — this is the paper trail that lets finance and legal sign off
  • Position agents as cost-per-unit reducers on existing workflows, not as headcount replacements (that framing kills procurement cycles)

Why it matters for PMs: Most AI ROI conversations stall because PMs come in with "here's how much the API costs and here's the output quality score." This framework flips it: start with the business KPI, work backward to what the agent needs to do, and then instrument that workflow so you can show the number moving. For anyone in fintech or enterprise software trying to get continued investment in AI features post-pilot, this is the structure to steal.

Critical questions:

  • The framework works cleanly for process-replacement use cases (RFP, AML). What happens when the agent value is additive rather than substitutive — does the model break?
  • How do you handle ROI attribution when humans are still in the loop for most decisions? "The agent did 80% of the work" is hard to operationalize in a spreadsheet.
  • LangSmith + Pay-i is a specific stack. How much of this framework requires that observability layer to work, versus being implementable with lighter tooling?
  • What's the failure mode when the KPI improves but the agent is taking shortcuts that create downstream risk?

Action you could take today: Pull up the last AI feature you shipped or are building, and write down three specific business KPIs it connects to — not AI metrics, not engagement metrics, but the KPIs your finance team already tracks. If you can't name three, that's your gap before your next stakeholder meeting.

LangChain — LangSmith Engine: The Agent That Improves Agents#

Source: https://www.langchain.com/blog/how-we-built-langsmith-engine-our-agent-for-improving-agents Credibility: High (first-party technical post from LangChain team, covers production architecture)

What happened: LangChain shipped LangSmith Engine — an agent that watches your other agents fail and proposes fixes. It analyzes traces at scale, identifies recurring failure patterns, groups them into actionable issues, and then suggests both evaluators (so you can detect the failure class automatically) and code-level fixes. It's essentially automated QA for LLM systems, closing the loop that most teams are doing manually in spreadsheets.

Key technical details:

  • Ingests production traces and clusters failures by type (not just by error code)
  • Converts recurring failure clusters into "issues" — persistent, trackable, prioritizable
  • Proposes evaluator logic that can detect the issue class automatically going forward
  • Also suggests code changes or prompt modifications to address the root cause
  • Designed to run on LangSmith's observability layer, which means it needs trace data to work

Why it matters for PMs: The biggest practical problem in shipping AI features is that the QA loop is manual and slow. You watch the logs, someone tags failures, an engineer figures out what's wrong, someone writes a new evaluator, repeat. LangSmith Engine is a bet that the agent can do most of that triage work. If it works, the feedback loop for AI features gets dramatically shorter — which directly affects how fast you can iterate on quality. This is relevant to the open question of whether AI changes product development velocity, and here the answer looks like: yes, specifically at the QA and debugging layer.

Critical questions:

  • How good is the failure clustering? Grouping failures by type is the hard part — does the agent understand semantic failure modes or just surface-level error patterns?
  • What's the false positive rate on proposed fixes? If the agent suggests code changes that make things worse, that's a new category of risk.
  • Does this work on smaller trace volumes, or does it require enterprise-scale data to find meaningful patterns?
  • Who owns the proposed evaluators and fixes — the agent team, the PM, someone else?

Action you could take today: If your team ships any LLM-powered feature, ask how failures are currently being triaged and categorized. If the answer is "someone reads the logs," you have a manual bottleneck that tools like this are designed to replace — and that context will help you evaluate whether LangSmith Engine is worth piloting.

Pieter Levels — More Apps, Less Sales#

Source: https://levels.io/more-apps-less-sales-than-ever Credibility: Medium (independent builder with direct revenue visibility into his own products; extrapolating from personal data to a broader claim)

What happened: Pieter Levels posted that the market for indie apps is being compressed from both sides: supply is exploding because AI makes building trivially easy, but demand isn't keeping up. His read is that we're in a paradox of choice problem at scale — more apps than ever, but users aren't buying more, and discovery is getting harder, not easier. He's seeing this in his own portfolio.

Key patterns:

  • AI lowered the cost of building to near zero, which flooded the market with competition
  • App stores and web discovery haven't gotten better at surface area, so discoverability is declining per product
  • Users aren't increasing their app spend to match supply growth — the pie isn't growing proportionally
  • The implication: differentiation and distribution matter more than ever, not less

Why it matters for PMs: This is the other side of the "anyone can build with AI" narrative that the vibe-coding conversation has been missing. Yes, you can ship faster. But so can everyone else, and the market for apps isn't infinite. For product teams inside larger companies, this is a signal that standalone AI features built to "ship something with AI" are increasingly undifferentiated — the moat has to come from distribution, data, or a workflow lock-in that a solo builder can't replicate. For teams evaluating whether to ship a standalone AI tool versus embedding AI deeper into an existing product, this post is evidence for the embedding strategy.

Critical questions:

  • Is this specific to the indie/consumer micro-app segment, or does it apply to B2B SaaS too?
  • Is this a temporary saturation phase before new discovery mechanisms emerge (AI-powered app recommendations, agent-driven installs), or a structural shift?
  • What does "sales going down" mean in this context — revenue, trial conversions, new user growth? The mechanism matters for what the fix would be.
  • If distribution is the moat, how do smaller teams actually compete on distribution against incumbents?

Action you could take today: Look at a feature or product your team is considering building as a standalone tool. Run it through the question: "Could a solo founder with Claude Code build a functional version of this in a weekend?" If yes, your moat question just got more urgent — figure out what makes your version defensible before you greenlight it.

OpenAI — Loveholidays and the Codex "Everyone Is a Builder" Playbook#

Source: https://openai.com/index/loveholidays Credibility: Medium (first-party OpenAI case study; customer-reported outcomes, not independently verified)

What happened: OpenAI published a case study on how UK travel company loveholidays used Codex to extend software development access to non-engineering teams. The headline claim: teams that couldn't previously build anything are now turning ideas into working products. It's OpenAI's version of the "Codex democratizes building" story, told through a real enterprise customer.

Key patterns:

  • Codex is being positioned as a way to close the gap between idea-havers and builders inside companies, not just accelerate existing engineers
  • loveholidays is using it to let product and operations teams ship lightweight internal tools without engineering queues
  • The framing is "everyone is a builder" — which is the same narrative Replit, Lovable, and Cursor have been pushing, now with an OpenAI enterprise wrapper
  • This connects to how AI changes the PM role: if non-engineers can ship tools, the PM becomes the spec-writer and prioritizer for a much larger pool of builders

Why it matters for PMs: The "democratize building" narrative is real but the case study leaves out the hard parts — maintenance, security review, integration debt. What's actually useful here is the implication for how product teams structure work: if Codex or equivalent tools are reducing the engineering queue for lightweight internal tools, PMs need to figure out where they add value in a world where the constraint is no longer "can we build this" but "should we build this and who maintains it."

Critical questions:

  • What happens to the tools loveholidays' non-engineers built when they break or need updates? Who owns that?
  • Is this primarily valuable for internal tooling, or does it extend to customer-facing products?
  • How does this change the PM-engineering relationship when the PM can ship without the engineer?

Action you could take today: Identify one internal tool or report your team has been waiting on engineering to build. Spend 30 minutes trying to build a rough version with Codex or Cursor — not to ship it, but to calibrate your team's actual leverage on this kind of work.

Quick Hits#

  • Lenny Rachitsky: Ryan Carson ($20K on Devin in a month) episode now has a companion newsletter post — runs 15 concurrent agents for engineering, customer success, and investor updates, uses a handwritten list to track them. Worth reading for the operational texture of what "running agents at scale" actually looks like day-to-day. (2026-08-24): https://www.lennysnewsletter.com/p/i-spent-20000-on-devin-in-a-month

  • Vercel: Shipped "Run SDK" — secure eval tooling for agents, positioned as a way to test agent behavior before production. Released alongside a post called "The end of credential sprawl for agents." Both are worth reading if you're building agent infrastructure. (2026-08-25): https://vercel.com/blog/introducing-run

  • Aravind Srinivas: Announced Perplexity x Intel collaboration bringing local models and hybrid inference to Intel Ultra Series 3 laptops. This is the on-device AI distribution story getting real — Perplexity embedding into hardware, not just app stores. (2026-08-26): https://x.com/AravSrinivas

  • LangChain: Shipped "Wiki Memory" — a file-based memory pattern for agents that compresses conversation history and decisions into a persistent knowledge base. Positioned as different from RAG (it's write-optimized, not retrieval-optimized). Concrete enough to be worth a look if you're building memory into agents. (2026-08-26): https://www.langchain.com/blog/wiki-memory

  • OpenAI: Shipped an Admin plugin for ChatGPT Work and Codex — lets admins analyze workspace usage, manage members and permissions, and adjust limits conversationally. The enterprise management layer is getting more capable. (2026-08-25): https://openai.com/index/introducing-admin-plugin

The Thread#

The moat question is getting sharper. Three threads this week point at the same tension: Pieter Levels watching the indie app market get flooded by AI-built tools, LangChain publishing ROI frameworks that help enterprise teams justify keeping their AI investments, and OpenAI running "democratize building" case studies for enterprise customers. The pressure is coming from both ends — the bar to ship dropped to near zero, which means differentiation can't come from building capability anymore. The teams winning are the ones with distribution (Perplexity on Intel hardware), workflow lock-in (LangSmith's observability layer), or data advantages that a weekend builder can't replicate.

Sit With This#

Pieter Levels is seeing something real: more apps being built than ever, but sales per app going down. The cost to ship dropped to near zero, and so did the barrier to competition.

For your product: What would a solo founder with Claude Code and a weekend build that competes directly with your current AI feature? If they can replicate the core value, what's the one thing they can't replicate — and is that thing actually in your product today?