5 Rules for Building AI Agents That Work in Production | Nan Yu & Jacob Shumway
A behind-the-scenes look at building a production AI agent end to end, including how to give agents tools to find the context they need and run evals to measure output quality.
Dear subscribers,
Today, I want to share a new episode with Nan Yu and Jacob Shumway.
Nan and Jacob built Linear Agent, which has transformed how the $1.25B company gets work done. In this episode, they walk through how they built an AI agent from the initial memo to launch, including how to give it tools to find the context it needs and how to design evals that make the agent more reliable. If you want a concrete, behind-the-scenes example of how to build a production agent end to end, this episode is a must-watch.
Watch now on YouTube, Apple, and Spotify.
We talked about:
(00:00) What’s holding AI agents back as models improve?
(00:51) Three things every good agent needs
(03:00) Demo: The internal memo that started Linear Agent
(09:50) The biggest mistake to avoid when building agents
(11:29) Demo: From Slack to code in 6 minutes with the agent
(17:01) When prototyping, “use the biggest model until it works”
(24:02) Give agents tools to find context, not more context
(34:07) The first thing to do before building an agent
I’m proud to partner with Oceans Talent
I’ve been working with Char, my assistant from Oceans Talent, for about six months. What stands out about his work is that he doesn’t just use AI tools. Instead, he actively builds and maintains AI workflows to help me run podcast post-production, draft show notes, make video clips, and more.
If you want to hire AI-fluent operators who can help you take back your time at 70% less than the cost of a US hire, book a call at oceanstalent.com/peter.
Top 10 takeaways I learned from this episode
The first step to building a good agent is to map the actual workflow. Identify where work starts, which systems hold context, what actions complete the job, what “done” means, and where a person needs to review. If work starts in Slack, make Slack the on-ramp instead of forcing users into a separate chatbot.
Give agents tools to find context instead of stuffing context into the prompt. Jacob’s advice: “Give it as little instruction as possible, to be honest. Give it the tools to load context. Don’t give it context.” Start small, then let the agent pull only the instructions and data it needs for the current task.