Home
Mar 22, 2026
View All

Stripe's A/B Test Data, Shreya Shankar on Claude Code vs Cursor, and Andrew Chen's Spreadsheet Thesis

·2 underrepresented voices

The Short Version#

Stripe published actual A/B test results on Adaptive Pricing across 1.5 million checkout sessions — rare, rigorous, and the evidentiary standard every AI feature claim should be held to. Meanwhile, Shreya Shankar's comparison of Claude Code vs Cursor agent mode surfaces a concrete architectural insight about why terminal-based agents outperform IDE-embedded ones, and Andrew Chen's spreadsheet thesis is the sharpest product strategy signal of the week: if code generation keeps improving, any workflow currently modeled as a spreadsheet is a product opportunity.

Stripe — A/B Testing Adaptive Pricing Across 1.5M Checkout Sessions#

Source: https://stripe.com/blog/product Credibility: High (first-party product post, authored by Sahil Bhagat, Product Manager at Adaptive Pricing, and Danielle Carollo, Data Scientist — a PM and data scientist co-authoring a methodology post is a strong credibility signal)

What happened: Stripe published the results of a production A/B test for Adaptive Pricing — their feature that dynamically adjusts subscription pricing based on user context — run across 1.5 million checkout sessions. This is not a launch announcement or a feature explainer. It's a methodology post that shows how Stripe designed, ran, and interpreted a large-scale experiment on a live pricing feature. The post was published March 19, authored by the PM and data scientist who own the feature.

Key details from the post:

  • 1.5 million real checkout sessions — not a lab study, not simulated traffic, not a cherry-picked cohort
  • Adaptive Pricing uses signals from user context to determine optimal pricing at checkout, which means the test was measuring a behavioral intervention at the moment of purchase decision
  • The post describes the experimental design, the treatment and control conditions, and how the team interpreted results — this is a complete methodology disclosure, not just a "results" announcement
  • Sahil Bhagat's PM byline on a data methodology post is notable: this is a PM who owns the measurement framework, not just the feature

Why it matters for PMs: Most "AI improves conversion" claims are vibes. This is data. Stripe ran a real controlled experiment at scale, published the methodology, and let the numbers speak. That's rare in fintech, and rarer still in AI feature development where teams routinely declare wins based on early adoption curves or anecdote. The PM-authored methodology post is also a model worth copying: when your PM can explain the experimental design, not just the headline metric, it means the team actually understood what they were testing. For PMs building AI features in fintech, e-commerce, or any checkout-adjacent context, this post is the template your stakeholders should be asking for. "We think this AI feature improves conversion" is not enough. "We ran this test at this scale with this methodology and saw this result" is.

Critical questions:

  • What was the actual effect size? The data shows a statistically significant result, but magnitude matters — was Adaptive Pricing moving conversion by 0.2% or 2%? The post summary doesn't surface the number.
  • How does Stripe handle the ethical dimension of dynamic pricing at checkout — are users in different treatment arms paying different prices for the same subscription? If so, how is that disclosed?
  • Was the experiment run across all geographies and plan types, or in a controlled subset? Selection criteria determine how generalizable the results are.
  • What's the latency of Adaptive Pricing — does dynamic pricing require a real-time inference call that affects checkout load time, and was that a confound in the experiment?

Action you could take today: If your team is running or planning an AI feature experiment, audit the experimental design: do you have a control group, a treatment group, a defined primary metric, and a pre-specified sample size? If any of those are missing, you don't have an A/B test — you have an observation. Use the Stripe methodology post as a checklist before you declare your next AI feature a success.

Shreya Shankar — Claude Code vs. Cursor Agent Mode#

Source: https://x.com/sh_reya/status/1956830797543166121 Credibility: High (Shreya Shankar is a researcher and practitioner in ML engineering and AI product development; this is a direct practitioner comparison from a tracked person, published March 22, 2026)

What happened: Shreya Shankar — ML researcher and one of the more technically rigorous voices in applied AI — posted a direct comparison of Claude Code (terminal-based) versus Cursor's agent mode, concluding that Claude Code in the terminal provides a superior experience. The comparison is specifically about prompt writing behavior and workflow efficiency — not benchmark scores or task completion rates, but the actual experience of working with each tool across a real workflow.

Key insight from the post:

  • Claude Code in the terminal prompts Shankar to think more carefully about what she's asking, while Cursor agent mode encourages more ambient, less deliberate prompting
  • The terminal interface creates a forcing function: you have to be specific, because there's no ambient IDE context to fill in the gaps
  • Cursor's integration into the editor lowers the friction of prompting — which sounds like a benefit, but can lead to vague requests that produce lower-quality outputs
  • The implication: the interface model of an AI coding tool shapes how you think about the task, not just how you execute it

Why it matters for PMs: This is a product design insight disguised as a tool comparison. The interface determines the quality of the interaction — a more opinionated, friction-bearing interface (Claude Code terminal) can produce better outputs than a smoother, more ambient one (Cursor agent mode), because it forces users to articulate their intent more precisely. This maps directly to a recurring product design question: when should you reduce friction, and when should intentional friction improve quality? For PMs designing AI features that require users to specify intent (research assistants, code agents, writing tools), Shankar's observation is a concrete signal: the interface that makes prompting feel easier may not be the one that produces the best results. There's a real design tradeoff here between accessibility and output quality.

Critical questions:

  • Does the terminal-vs-IDE distinction hold across different task types? For exploratory work (debugging, refactoring), the ambient IDE context might be an advantage. For greenfield work, the terminal may win. The comparison likely varies by workflow.
  • Is this an argument for a specific interface (terminal) or a specific interaction model (deliberate, explicit prompting)? If the latter, could Cursor replicate the benefit by making prompting more deliberate within the IDE?
  • How does context window usage differ between the two? Claude Code in the terminal doesn't have access to the full IDE state, which might produce cleaner context and better outputs — or might miss relevant code.
  • Is Shankar's experience generalizable to less technically experienced users, or does the terminal's benefits require a user who already knows what they want?

Action you could take today: If your team uses Cursor agent mode, run a one-day experiment: pick one non-trivial coding task and complete it twice — once in Cursor agent mode, once with Claude Code in the terminal. Track not just the output quality, but how you formulated the request in each interface. Did the terminal make you more specific? That delta in prompt quality is the actual product signal.

Andrew Chen — AI Code Generation Will Make Spreadsheets Obsolete#

Source: https://x.com/andrewchen (post, March 18, 2026) Credibility: High (Andrew Chen is a general partner at a16z and a tracked thinker on product growth and market dynamics; this is a direct statement of strategic thesis, not speculation)

What happened: Andrew Chen posted a sharp product strategy thesis: AI-powered code generation will render spreadsheets effectively obsolete as a modeling tool. The core argument: "AI code gen means that anything that is currently modeled as a spreadsheet is better modeled in code. You get all the advantages of software — libraries, open source, AI, all the complexity and expressiveness." The post is brief but carries significant product strategy implications.

Why this is more than a hot take:

  • Spreadsheets persist not because they're the best tool for modeling, but because they're the accessible tool — no coding required, no deployment, no maintenance burden
  • If AI code generation collapses the accessibility barrier, the spreadsheet's core advantage (no-code, instantly usable) evaporates
  • What replaces spreadsheets isn't Excel with AI features — it's code that a non-developer can generate, run, and modify via AI
  • This maps to a broader pattern: AI is eroding the category of "things that require a specialist to build," and spreadsheets are one of the largest categories of workaround software

Why it matters for PMs: This is a direct product opportunity signal. Every SaaS product that competes with a spreadsheet workflow — financial modeling, data tracking, operational reporting, inventory management, capacity planning — should be asking: if AI makes custom code as accessible as a spreadsheet, what's our defensible position? The answer is probably not "we're better than a spreadsheet." It's "we provide the infrastructure, collaboration, governance, and reliability that a generated script can't." For PMs at companies like Notion, Airtable, or any vertical SaaS competing against Excel, this thesis deserves a dedicated strategy session. And for PMs at fintech companies (relevant to this digest): your users are modeling their businesses in spreadsheets. That's both a threat and an opportunity.

Critical questions:

  • What's the actual adoption curve here? Code generation is improving fast, but the gap between "can generate code" and "can maintain, debug, and share code with a team" is still significant.
  • Does this argument apply uniformly across user types? A finance director at a mid-market company vs. an analyst at a startup have very different risk tolerances for "AI-generated code" running their financial model.
  • What does version control, auditability, and compliance look like for AI-generated code that replaces a spreadsheet? Those are exactly the properties spreadsheets have (everything in one file, easily shared, trivially auditable) that code does not.
  • The Runway company blog in today's data has three posts about spreadsheet alternatives and financial modeling. Is that a signal they're already repositioning against this thesis, or just content marketing?

Action you could take today: List the top three workflows your users currently handle in spreadsheets (check support tickets, user interviews, or usage analytics for "export to CSV" or "export to Excel" signals). For each one, ask: if a user could generate a custom script in 10 minutes that did this better, would they? That's the vulnerability map for your product's current spreadsheet-adjacent value.

Quick Hits#

The Thread#

The evidentiary bar for AI feature claims is being set — and most teams aren't meeting it. Stripe's A/B test post is the week's clearest example: 1.5 million sessions, controlled methodology, PM and data scientist co-authoring the writeup. Shreya Shankar's Claude Code vs Cursor comparison adds the practitioner layer — a rigorous thinker doing her own controlled experiment of one. Andrew Chen's spreadsheet thesis is the strategic frame. Together they point to a discipline question that's becoming harder to avoid: are you shipping AI features because you tested them, or because they felt right? The teams that will own their categories in two years are the ones building measurement infrastructure now, not retrofitting it after launch.

Sit With This#

Shreya Shankar found that Claude Code in the terminal produced better outputs than Cursor agent mode — not because the model was better, but because the terminal interface forced more deliberate, specific prompting. The friction was the feature.

For your AI product: You almost certainly have a user-facing AI interaction where you've been optimizing to reduce friction — faster prompting, more ambient context, fewer required inputs. Is there a case where that friction reduction is actually degrading output quality because it's letting users be vague? Pick one interaction and ask: what would happen if we required users to be more specific before the AI responds?