#002 - Luke Frake, Spotify
In this conversation, Alexandre Suon and Luke Frake discuss the topic of experimentation strategy. They explore whether it is better to focus on building as many tests as possible or to focus on building the best test possible.
Summary
In this conversation, Alexandre Suon and Luke Frake discuss the topic of experimentation strategy. They explore whether it is better to focus on building as many tests as possible or to focus on building the best test possible. Luke shares his journey into experimentation and emphasizes the importance of both speed and quality in running experiments. They discuss the challenges of scaling experimentation and the need for standardized documentation and templates. They also touch on the topic of tools for experimentation and the factors to consider when determining the maximum number of tests a company can run. The conversation explores the role of experimentation in balancing speed and quality. They discuss the importance of having a north star metric and how impact is the combination of speed and quality. They also touch on the need to justify the return on investment of the experimentation program and the importance of learning from both wins and losses. They share tips on sharing experiment results within the company, such as gamification and making the program searchable. They emphasize the need for flexibility in focusing on speed or quality depending on the context and the overall goal of increasing impact.
Keywords
experimentation strategy, building tests, speed vs quality, scaling experimentation, documentation, templates, tools for experimentation, maximum number of tests, experimentation, speed, quality, impact, return on investment, learning, justification, sharing results, gamification, program management, flexibility
Takeaways
Experimentation strategy should focus on both speed and quality.
Starting with a few tests and gradually scaling allows for learning and improvement.
Standardized documentation and templates are important for maintaining quality as experimentation scales.
Tools like Airtable and Conversion.ai can be helpful for organizing and managing experimentation.
The maximum number of tests a company can run depends on factors like traffic and sample size. Experimentation is about balancing speed and quality to increase impact.
Having a north star metric and measuring impact is crucial.
Learning from both wins and losses is important for growth.
Sharing experiment results within the company can be done through gamification and making the program searchable.
Flexibility is key in focusing on speed or quality depending on the context.
Titles
Determining the Maximum Number of Tests
Scaling Experimentation: Challenges and Solutions Flexibility in Focusing on Speed or Quality
Balancing Speed and Quality in Experimentation
Sound Bites
"The answer to the question of speed vs quality is both."
"Start with something, don't go for perfection from the beginning."
"Templatizing the hypothesis is really important."
"Experimentation is a great tool to increase the risk you're taking."
"The outcome of experimentation should be impact and change."
"All experiments can be considered learning, but not at the expense of impact."
Chapters
00:00 Introduction and Small Talk
01:14 Background and Experience in Experimentation
08:30 Approach to Experimentation in New Companies
13:00 The Importance of Standardized Documentation and Templates
22:21 Determining the Maximum Number of Tests
23:32 Balancing Speed and Quality
27:05 Measuring Impact and Learning
35:35 Sharing Experiment Results
45:20 The Role of Experimentation
