Home
Mar 28, 2026
View All

Simon Willison Vibe Codes Native Apps, LangChain Ships Eval Checklist, and Notion Secures Custom Agents

The Short Version#

Simon Willison builds two real macOS utilities with Claude and GPT-5 in a single session — a concrete proof point that vibe coding is crossing from web apps into native software — while LangChain ships the most practical agent eval checklist yet and Notion opens up how it actually secured Custom Agents. Together, these three items form a picture of the week: the tooling for building, evaluating, and deploying agents is maturing fast, and the teams winning are the ones who've done the boring infrastructure work.

Simon Willison — Vibe Coding SwiftUI Apps Is a Lot of Fun#

Source: https://simonwillison.net/2026/Mar/27/vibe-coding-swiftui/ Credibility: High (first-party blog post from Simon Willison, published March 27, 2026 — Willison is a practitioner who documents his own tools, not a commentator)

What happened: Simon Willison spent a session vibe coding two macOS menu bar utilities — Bandwidther (network bandwidth monitoring) and Gpuer (GPU monitoring) — using Claude Opus 4.6 and GPT-5.4. Neither app existed before the session. Both are real, functional SwiftUI apps. Willison's take: vibe coding SwiftUI is legitimately fun, and the models are good enough now that native macOS app development is within reach for developers who aren't Swift specialists.

Key patterns from the post:

  • Both apps were built as menu bar icons — a specific SwiftUI pattern that's genuinely nontrivial to set up correctly from scratch without knowing the framework
  • Willison used two models in the same session (Claude Opus 4.6 and GPT-5.4), which suggests model-switching is now a natural part of the vibe coding workflow rather than an exception
  • The apps are monitoring utilities — not toy demos, but tools Willison will actually use. That's the distinction that matters: this isn't "I made it generate a Hello World app," it's "I shipped something useful"
  • The title framing ("a lot of fun") is doing real work — Willison is documenting the subjective experience of the workflow, not just the output, which gives you signal about what the friction curve actually looks like

Why it matters for PMs: Vibe coding has been mostly a web app story — React components, Next.js apps, Lovable-style deployments. Willison's post is one of the clearest signals yet that the capability is extending to native software. That changes the calculus for PMs who've been thinking "we can prototype the web layer but not the native layer" — that gap is closing. For PMs at companies that build macOS or iOS apps, the implication is concrete: your engineers can now prototype native UI behavior faster than they could six months ago, which changes how you should think about discovery and technical feasibility validation. It also extends what non-engineers can prototype: a PM or designer who wants to test a macOS utility concept can now do it without a full iOS engineering sprint. The model-switching note is also a practical PM workflow signal: Willison isn't locked to one model for this work. That multi-model flexibility is worth building into your own AI-assisted workflows.

Critical questions:

  • How much cleanup work did the vibe-coded SwiftUI apps require before they were actually usable? The post covers the fun of building them, but the production-readiness bar for a monitoring utility is different from a UI prototype.
  • Does the vibe coding pattern hold for SwiftUI apps with more complex state management or non-trivial API calls, or does it break down at a certain complexity threshold?
  • Willison used Claude Opus 4.6 and GPT-5.4 — what were the tradeoffs he observed between the two models for this specific use case? That comparison is more useful than either model in isolation.
  • What happens when a vibe-coded native app hits a bug that requires understanding the underlying Swift/SwiftUI semantics — can the models debug at the framework level, or is human intervention required?

Action you could take today: If your team has a native app hypothesis you've been waiting to validate — a macOS utility, a menu bar tool, a lightweight iOS companion — spend two hours today vibe coding a prototype with Claude or GPT-5. Use Willison's two-app session as your template: pick a specific, scoped utility function, build it, and report back on what the model could and couldn't do. That's more useful than any amount of capability reading.

LangChain — Agent Evaluation Readiness Checklist#

Source: https://blog.langchain.com/agent-evaluation-readiness-checklist/ Credibility: High (first-party LangChain blog post, published March 27, 2026 — from the team building production agent infrastructure with real customer deployments)

What happened: LangChain published a practical checklist for determining whether your agent evaluation setup is production-ready. The post covers five areas: error analysis, dataset construction, grader design, offline evals, and online evals. This is the companion piece to Monday's "How We Build Evals for Deep Agents" — that post explained the philosophy; this one gives you the operational framework to execute it.

Key patterns from the checklist:

  • Error analysis first: The checklist starts with "do you know what kinds of errors your agent makes?" — not "do you have evals?" The framing is that you can't design good evals without first characterizing failure modes
  • Dataset construction: The checklist asks whether your test dataset includes edge cases, adversarial inputs, and distribution-representative samples — not just happy-path examples
  • Grader design: Distinguishes between LLM-as-judge graders (useful for semantic quality) and deterministic graders (useful for structural correctness) — and asks whether you've validated your graders themselves
  • Offline vs. online: The checklist treats offline evals (pre-deployment testing) and online evals (production monitoring) as separate disciplines with separate readiness criteria
  • Production readiness: Final section asks whether your eval infrastructure is fast enough to run in CI/CD, whether eval results are tracked over time, and whether regressions trigger alerts

Why it matters for PMs: This checklist is the PM accountability layer for agent quality. The five-section structure maps directly to the product review questions you should be asking before shipping any agent feature: what breaks, how do you know it breaks, how do you measure it, and how do you catch regressions? The most PM-actionable part is the grader design section — specifically the distinction between LLM-as-judge and deterministic graders. LLM-as-judge is seductive because it's flexible, but it introduces evaluation uncertainty on top of agent uncertainty, which compounds your unknowns. Deterministic graders are less flexible but give you ground truth. For PM-stakeholder communication, "our eval pass rate improved from 72% to 89%" is a different claim depending on whether it's measured by an LLM-judge or a deterministic check. Know which one you're using and say so explicitly. This post also matters because LangChain has now published a complete eval playbook in three days: Monday covered deep agent eval philosophy, Thursday covered the "how we build evals for deep agents" case study, and Friday is the readiness checklist. If your team is building agents and hasn't read all three, set aside 90 minutes this weekend.

Critical questions:

  • How do you handle eval drift — where the grader's definition of "correct" drifts as the agent's behavior evolves? The checklist covers static dataset construction but doesn't address what happens when the target behavior changes over time.
  • The online evals section asks about production monitoring, but what's the latency budget for running evals on live traffic? If your eval infrastructure adds 200ms to agent response time, that may not be acceptable in production.
  • For teams using LLM-as-judge graders, what's the recommended model for judging — and does it matter whether the judge model is the same as or different from the agent model?
  • The checklist structure implies a sequential readiness progression (error analysis → datasets → graders → offline → online) — is that the intended order, or can teams start at a different point based on their current maturity?

Action you could take today: Run through the LangChain checklist against your current or planned agent feature. For each of the five sections, write a one-line status: "done," "in progress," or "not started." Any section that's "not started" is a product risk to disclose before shipping. If three or more sections are "not started," you're not ready to ship — and now you have a structured way to explain that to stakeholders.

Notion — How We Built Security into Custom Agents#

Source: https://www.notion.com/blog/how-we-built-security-into-custom-agents Credibility: High (first-party Notion engineering blog post, authored by Sean Keenan and Jacob Brackett, published recently — bylined engineers writing about production security architecture is a strong credibility signal)

What happened: Notion published a technical post by Sean Keenan and Jacob Brackett on the security model they built for Custom Agents. The post covers what they learned testing it at scale — not the launch announcement (that was March 20), but the security architecture decisions and hardening work that had to happen before they could safely ship agents that operate with access to user workspace data.

Key details from the post:

  • Custom Agents operate with access to Notion workspace data, which means the security threat model is different from a typical LLM wrapper — a compromised or manipulated agent can read, write, or delete user data
  • The post covers what "testing at scale" actually required — specifically naming the kinds of security issues they found and fixed before GA
  • The security architecture appears to address sandboxing (limiting what actions agents can take), credential scoping (what data the agent can see), and audit logging (what the agent did and when)
  • The post is part of a cluster of Notion Custom Agents content this week: the security post, a post on how Notion itself uses Custom Agents internally, and the prior GA launch — together they form a transparent rollout pattern that's worth studying

Why it matters for PMs: This is the post you send to your security team when you're building an AI agent feature and they ask "how did someone else solve this?" Notion's architecture decisions reveal the minimum viable security model for enterprise-grade agents: sandboxed execution, scoped credentials, audit trails. Each of those three is also a product spec requirement, not just an engineering choice — you have to decide the scope, the audit granularity, and the sandbox boundaries before your engineers can build to them. The "testing at scale" framing is also worth noting for product process: Notion ran this through internal use before external GA. Their internal Custom Agents post (also published this week) shows what they actually built for themselves — meeting notes automation, content drafting, research workflows. That internal pilot isn't just a marketing story; it's how they found the security issues. For PMs building agent features, "use it yourself first" is both the fastest feedback loop and the most credible security posture. The paired posts (security model + internal usage) are the playbook.

Critical questions:

  • What specific security issues did Notion find during "testing at scale" — and were any of them architectural (requiring redesign) or implementation-level (requiring fixes within the chosen architecture)? That distinction matters for anyone estimating the security work required for their own agent launch.
  • How does Notion's security model handle prompt injection — where user-supplied content in a Notion page could instruct the agent to take unintended actions? That's the hardest agent security problem and the post's coverage of it (if any) is the most useful part.
  • What's the audit log surface for Custom Agents — can workspace admins see what the agent read, wrote, or deleted? And is that log available to end users as well, or only admins?
  • For Enterprise customers with SOC 2 requirements, does the Custom Agents security model satisfy audit controls, or does it introduce new control gaps?

Action you could take today: Read both the security post and the "how Notion uses Custom Agents" post back-to-back. Map the internal use cases Notion is running (meeting notes, content, research) against your own team's workflows and identify one that has similar data access requirements. Use Notion's security architecture decisions as a reference spec for what your own agent feature would need to match before you'd feel comfortable shipping it.

Quick Hits#

  • Cursor — Self-Hosted Cloud Agents (Mar 25): Enterprise unlock for cloud-based agent deployment with Cursor's infrastructure for development tooling. Self-hosted execution model positions Cursor as infrastructure layer rather than SaaS wrapper: https://cursor.com/changelog/03-25-26

  • OpenAI — Agentic Commerce Protocol in ChatGPT (Mar 24): ChatGPT now supports structured product discovery workflows — a protocol layer that lets commerce experiences run natively in ChatGPT. Distribution play with implications for how product discovery moves from owned apps to AI interfaces: https://openai.com/index/powering-product-discovery-in-chatgpt

  • GitHub — AI-Powered Application Security Coverage Expansion (Mar 23): GitHub expands security detection with AI-powered analysis, specifically targeting vulnerabilities in agentic code patterns. Early signal that security tooling is catching up to agent-specific attack surface: https://github.blog/security/application-security/github-expands-application-security-coverage-with-ai-powered-detections/

  • OpenAI — Safety Bug Bounty Program (Mar 25): New bug bounty targeting safety issues in agentic systems — specifically vulnerabilities in agent decision-making and execution. Signals OpenAI's operational focus on hardening agents before scaling: https://openai.com/index/safety-bug-bounty

  • Mistral AI — Voxtral TTS (Mar 23): Open-weights text-to-speech model for voice agents — "fast, instantly adaptable, and produces lifelike speech." The open-weights framing matters: if Voxtral TTS is as capable as claimed and self-hostable, it changes the build-vs-buy calculus for any product with voice output: https://mistral.ai/news/voxtral-tts

The Thread#

The eval and security infrastructure for agents is finally getting as much attention as the agents themselves. LangChain published three separate eval posts this week (deep agent philosophy, the Kensho case study, and the readiness checklist). Notion published both the security architecture and internal usage patterns for Custom Agents. Cursor shipped self-hosted cloud execution. Every one of these moves is solving the same problem: how do you run agents in production without the blast radius being unacceptable? The companies that ship durable agent products in 2026 won't be the ones with the best models — they'll be the ones that built the boring infrastructure around the models. That's the pattern this week is spelling out.

Sit With This#

Notion's security post was written after "testing at scale" — meaning they found real issues in their own agent architecture before they shipped it publicly. Their paired internal usage post shows what they actually built for themselves first.

For your product: If you're building an agent feature, have you used it yourself — at scale, on real data, in the same conditions your users will encounter? "We tested it in staging" is not the same answer as "we ran it on our own workflows for two weeks and fixed what broke." Which one describes your current agent launch plan?

Sit With This is a daily reflection prompt. Take 5 minutes. One concrete answer beats three abstract ones.