Miqdad Jaffer: Inside How Shopify Built Its Generative AI Products

A behind the scenes look at how great gen AI products are built

Level up your product and creator skills in just 5 min a week. Join 60,000+ readers:

---

Dear subscribers,

Today, I want to share an inside look at how Shopify builds gen AI products.

Miqdad Jaffer is a PM director who leads Shopify’s AI efforts. I’ve always admired the company’s craft-centric and meeting-adverse culture. Let’s dive into how Shopify:

1. Used its “get shit done” process to accelerate its AI efforts

2. Built Sidekick, its AI assistant for merchants

3. Prioritizes quality and craft across all products

Keep reading if you want an “explain like I’m five” guide about Shopify’s culture and how gen AI products are actually built.

---

How Shopify used its GSD process to accelerate its gen AI effort

Welcome Miqdad! How did you and the team start Shopify’s gen AI efforts?

Tobi (Shopify CEO & founder) kicks off our annual planning every year by writing a letter from a merchant. In last year’s letter, one theme was "Shopify keeps me on the cutting edge."

ChatGPT’s success made it clear to us that gen AI could help merchants stay on the cutting edge. For example, merchants often don’t have time to write great product descriptions for their items. Gen AI is a perfect fit for this use case.

So we quickly assembled a team to ship AI product descriptions for our Winter Edition 2022 release. Merchants loved it and we decided to double down on AI.

Shopify’s first gen-AI product was AI product descriptions

So you just picked a use case and shipped it instead of writing a big strategy doc?

Yes. We wanted to deliver value to merchants quickly.

We could’ve written a big strategy doc, but often it’s quicker to just ship something and have merchants provide real-time feedback as they adopt the product.

Shopify has a product development process called GSD or “Get Shit Done.” How did you navigate this process with AI product descriptions?

GSD has 5 stages, here’s how we navigated it with this product:

1. Proposal. We started by explaining the merchant problem, product principles, and our solution. Our main principle for AI product descriptions was to “let the merchant decide.” Merchants should control whether to accept AI’s suggestions.

2. Prototype. We then built a prototype quickly to get a sense of how the product might work. We quickly realized that AI needed detailed prompts to write great product descriptions. We used retrieval augmented generation to add more context (more on this in the next section). 

3. Build. We expanded our prototype to a full product and shared our progress with leadership along the way.

4. Release. We launched the product to a segment of our users. To get to market faster, our initial AI release was just for English-speaking merchants.

5. Results. We shared results and next steps (e.g., add multiple language support).

Instead of doing product reviews, I heard that Shopify PMs sometimes just share a short async video with leadership. Is this true?

Yes, it’s true!

I don’t think we had many sync product reviews for this product. It was almost all async.

1. We had a public Slack channel with leadership and other stakeholders.

2. Every few days, we would share a demo video and ask for feedback. 

3. Our VP Glen Coates would then make a video friction log about what he liked and didn’t like. We would then turn around a new demo in a few days.

When did you talk to real merchants and customers during the GSD process?

Merchants were involved throughout the process: 

1. Prototype. We shared mocks with merchants for feedback. 

2. Build. We generated actual product descriptions for select merchants.

Building with our merchants made it clear what good looks like and what our launch messaging should be.

After your success with AI product descriptions, you built a whole suite of gen AI products called Shopify Magic. How did you prioritize the jobs to be done?

We always start with our mission to make merchants successful. 

Next, we looked at the entire merchant journey to identify where gen AI could help the most. It quickly became clear that marketing (e.g., email subject lines, store copy & themes, SEO-optimized blog posts) was a big area of opportunity for gen AI.  

Did you have specific OKRs to meet for your gen AI efforts?

We didn’t have a goal like “reach X adoption in Y months.” 

Setting adoption goals for new products leads to growth hacks that aren’t sustainable.

We wanted to make sure that merchants were using AI because it solved their problems, not because of a growth hack. We did run an experiment to make sure that AI was at least neutral or positive to our company metrics.

---

A deep dive into how Shopify built Sidekick, its AI assistant for merchants

Tobi made a great video introducing Sidekick

Let’s cover Shopify Sidekick next. The product got great press, can you describe it?

Sidekick is our AI assistant that helps merchants grow their businesses. It can:

1. Fill out forms (e.g., “Create a student discount code for 10% off in June”)

2. Write content (e.g., “Write a blog post to promote my products”)

3. Run analytics (e.g. “What are my best-selling products?”)

4. Edit store themes

5. Put items on sale

I would love to understand in simple terms how a gen AI product like Sidekick is built. We can cover how to:

1. Personalize AI’s answers

2. Run an evaluation process

3. Trade-off quality and cost

But first, can you tell me how building a gen AI product like Sidekick is different from building a non-AI product?

With non-AI products, you set goals, define requirements, and then work through the edge cases. 

With a gen AI product, it’s very hard to predict every edge case. So you have to assume that everything can go wrong. For example, Sidekick might make mistakes or a merchant might ask for something that it can’t do. 

To deal with gen AI’s infinite edge case problem, define clear product principles early.

For example, the principle of “let the merchant decide” led us to design a UX where merchants could review and approve Sidekick’s suggestions. We don’t want Sidekick to put stuff on sale without the merchant even knowing about it.

What are the different layers of an AI product like Sidekick?

Sure, from bottom to top, there are 4 layers:

1. Model. This includes the LLM model and any adapters or fine-tuning on top of it.

2. API. This connects the UI to the model and handles chat history, errors, text streaming, and more.

3. UI. This lets merchants chat with the AI, submit feedback, and view errors.

4. Admin. This is how #1-3 integrates with Shopify systems like merchant data.

How to personalize AI’s answers

Many companies are building AI chatbots. How do you personalize Sidekick’s answers compared to a generic LLM like ChatGPT?

Retrieval augmented generation (source)

There are several ways to personalize AI’s answers, from least effort to most effort: