How to Run Experiments That Don't Suck
A practical guide on when to run A/B tests and how to make them actually useful
Level up your product and creator skills in just 5 min a week. Join 50,000+ readers:
Dear subscribers,
Today, I want to talk about how you can run experiments that don’t suck.
Eric Metelka is head of product for Eppo where he has a broad view of how fast growing companies run experiments. I spoke to Eric about:
When to run experiments
How to run experiments that don’t suck
How to introduce experimentation to your company
This post is brought to you by…Eppo
I've run 100s of experiments in my PM career so I know how painful it is to build a great experimentation platform in-house.
Eppo is the only 3rd party platform that's built on top of your company's data warehouse. This lets you track critical server-side metrics in real-time without having to share data externally or implement client events. Give it a try for free below!
When to run experiments
Welcome Eric! How do you decide if you should run an experiment or not?
You should run experiments if you need to…
Prove causality. We’ve all attended meetings where people question if a product moved a metric. An experiment removes this uncertainty.
Measure impact. For example, you may run an experiment that grows engagement by 3% but hurts revenue by 1%. Measure impact to make decisions.
You should not run experiments if you need to…
Make a decision quickly. Experiments usually take time and work to run. They shouldn’t be used to make every product decision.
Bet on a 0-1 product. If you’re building something new, you might have too few users or not enough data to run an experiment.
Do a big launch. If you need to make a big public launch, it’s hard to experiment.
As a rule of thumb, experiments are most useful when you have product market fit and scale. They’re great for optimizing key user flows like onboarding, engagement, and checkout.
How do you run an experiment?