Inside the AI Product Manager Role
Inside the AI Product Manager Role
Product leaders from OpenAI, Google, Meta, Figma, Roblox, and more reveal how to break into AI PM and lessons from crafting great AI products
This post was co-authored by Agnes Chu.
Dear subscribers,
Today, I want to share an inside look at the AI product manager role.
In this post, you’ll hear from AI PMs at OpenAI, Google, Meta, Figma, Roblox, Adobe, and more on how to break into AI PM and craft great AI products.
We asked each product leader just four questions:
What are your top learnings from building AI products?
How did you become an AI PM?
What skills do AI PMs need? How technical do you have to be?
How can someone become an AI PM?
Let’s dive into the answers (and memes).
What are your top learnings from building AI products?
David Kossnick (Senior Director of Product at Figma)
A prototype is worth a thousand mocks.
Plenty of exciting AI ideas die at the prototype stage because current AI models just can’t do the desired task well enough — it’s too slow, inconsistent, or hallucinate too much. Prototypes can identify these issues before you get too far into a project. Even if you can’t build a full prototype, at least prove in a model-prompting playground that the right inputs can get you the right output.
Create a data loop.
Whether through implicit signals like accept and reject rates or explicit thumbs up and down, make sure you’ve got a data loop flowing so that you can continue to improve quality after you ship.
Miqdad Jaffer (OpenAI Product Lead)
Getting your data in a good place is half the battle.
Start with a prototype, and don't expect that you’ll always get to production. Often, the customer problem can’t be solved due to current model limitations. In fact:
Expect and anticipate failure.
The effectiveness bar for a machine is always orders of magnitude greater than the same bar for a human. Dealing with that paradox is part of being an AI PM.
Sujoy Banerjee (Roblox Product Lead)
Invest in evals, focus on model capabilities, and treat labeled data like gold.
Invest in evals. Determine your desired outputs early so your engineers can evaluate progress and iterate quickly.
Focus on capabilities. UX is important, but without a capable model, it’ll always be challenging to build something amazing. The model’s capabilities can dictate what kind of UX you can support.
Labeled data is gold. Find ways to have user interactions reinforce your models, whether that's through implicit or explicit signals. For example, on Roblox Assistant, we look at thumbs-up rates, follow-up messages, and undo rates to understand which responses are high quality and which need improvement.
Marily Nika (Google Product Lead and instructor)
Understand that AI can deliver a different user experience each time.
AI’s probabilistic nature can make a user’s experience different every time they use a feature, which requires setting clear expectations about how the product will perform. This is most important when presenting to leadership; they must understand that AI might not always deliver the same experience. Also, make sure that you:
Foster an experimentation culture.
You cannot build in a silo and hope that your AI product will succeed. You have to be constantly talking to customers and testing in the market. Some specific advice:
Be scrappy: Use available resources efficiently to iterate quickly and learn.
Be precise about experiment outcomes: Clearly define what success looks like.
Time box experiments: Limit the duration of experiments to ensure timely insights and avoid over-investment in any single approach.
Remember that AI performance often drops when moving from experiments to production. To manage this, define a Minimum Viable Quality (MVQ) for your product. This ensures that you deliver a reliable user experience in the real world.
Sohrab Fathi (Meta PM)
Don’t just blindly apply AI to solve all your problems.
AI is a tool that doesn’t always offer the best ROI to solve user problems. For example, for search rankings or recommendations, we start with heuristic, rules-based models as a baseline. Honestly, simple rules-based models are often good enough.
Communicate risks with stakeholders proactively.
Since AI projects have a lot of uncertainty, it’s important to communicate risks with leadership early. That way, there are no surprises three months later when something didn’t happen. Try to especially over-communicate with AI/ML platform teams because those organizations often have different goals from product teams.
Hao Xu (Adobe PM)
Don’t underestimate user skepticism towards AI.
You need to talk to users constantly to validate your hypothesis. I realized that not all users (especially creatives) were excited about AI tools.
At Adobe, we launched six AI features. “Generate similar images” was very popular as it addressed the lack of control in AI. In contrast, “Text to image” was less popular since users could already do that with other tools. It’s about finding problems you’re uniquely positioned to solve for users.
Renee Park (Woven VP Product)
It can be frustrating when your AI product degrades due to a model change.
When LLM providers update their existing models, these changes can impact customers using your AI product. If consistency is mission-critical, you may consider using a model you can control or having a waterfall of replacement models.
Have multiple bets and prototype solutions early.
You won’t know if your model can support your use case until you prototype it. Often, the current tech just isn’t there to produce consistently valuable responses. So, pursue multiple bets and pick your winners.
How did you become an AI PM?
Miqdad Jaffer (OpenAI Product Lead)