#Special - Lee Carson, Comcast
In this conversation, Lee shares his journey in experimentation, starting from his background in graphic design and development to becoming a project manager. He discusses his experience at Marriott, where he implemented a successful experimentation program that doubled revenues.
Summary
In this conversation, Lee shares his journey in experimentation, starting from his background in graphic design and development to becoming a project manager. He discusses his experience at Marriott, where he implemented a successful experimentation program that doubled revenues. Lee explains the process of starting and scaling the program, collaborating with marketing teams, and combining experimentation with user research. He also highlights the competitive pressure in the travel industry and the benefits of personalization in experimentation. Finally, Lee talks about his transition to Comcast and the maturity of experimentation at the company. In this conversation, Lee discusses his experience in experimentation and the challenges he faces at Comcast. He highlights the importance of aligning experimentation with strategy and the need for management buy-in. Lee also talks about implementing server-side testing and the pitfalls and pains associated with it. He emphasizes the value of data analysis and the need for custom tools to facilitate advanced analysis. Lee shares his approach to experiment documentation and the importance of being able to look back at experiments. He concludes by defining experimentation as the rigorous application of curiosity.
Takeaways
Starting with a solid foundation of user insight and combining it with best practices can lead to successful experimentation.
Collaboration with marketing teams is crucial in convincing them to prioritize experimentation over pushing their own content.
Scaling an experimentation program requires hiring a diverse team with expertise in strategy, user research, and technical implementation.
Combining experimentation and user research allows for deeper analysis and insights.
In competitive industries like travel, rigorous experimentation is necessary to stay ahead.
Personalization can be effective in certain areas, such as service and recommendation customization.
Transitioning from a mature experimentation program to a new company requires adapting to different levels of maturity and stakeholder engagement. Aligning experimentation with strategy is crucial for success.
Implementing server-side testing can be challenging but offers valuable insights.
Custom analysis and data cleaning are essential for accurate experiment results.
Experiment documentation and the ability to look back at experiments are important for learning and improvement.
Experimentation is the rigorous application of curiosity.
Chapters
00:00 Introduction and Background
00:17 Getting into Experimentation
03:03 Transition to Product-based Experimentation
04:02 Starting and Scaling the Experimentation Program at Marriott
06:12 Collaborating with Marketing Teams
07:08 Growing the Experimentation Team
09:19 Combining Experimentation and User Research
10:06 Competitive Pressure in the Travel Industry
12:32 Benefits of Combining Experimentation and User Research
14:08 Personalization and Experimentation
19:12 Maturity of Experimentation at Comcast
19:16 Introduction and Background
20:12 Challenges and State of the Industry
20:42 Focus on Digital Channels
21:12 Multi-Channel Experimentation
21:31 Enabling Other Teams to Run Experiments
22:11 Aligning Experimentation with Strategy
22:23 Experimentation as a Glue Between Management and Operations
23:12 Implementing Server-Side Testing
24:15 Influencing the Product Roadmap
25:16 Improving Prototyping Capabilities
26:08 Importance of Management Buy-In
26:24 Challenges of Server-Side Testing
27:00 Implementing Basic Server-Side Testing
27:44 Analyzing Experiment Results
28:14 Importance of Data Cleaning and Custom Analysis
29:12 Testing on Chatbots
29:42 Market-Based Split Testing
30:08 Importance of Experiment Documentation
31:35 Looking Back at Experiments
32:37 The Value of Losing Ideas
36:16 Advice for Someone Starting in Experimentation
38:47 Experimentation Beyond Tools
40:33 Defining Experimentation
41:26 Experimentation as Rigorous Application of Curiosity
