#003 - Sadie Neve, Tesco
In this conversation, Alexandre Suon and Luke Frake discuss the topic of experimentation strategy. They explore whether it is better to focus on building as many tests as possible or to focus on building the best test possible.
Summary
Sadie Neve shares her journey into experimentation, starting with her background in psychology and stumbling upon CRO (conversion rate optimization). She emphasizes the importance of stakeholder buy-in and the challenges of starting an experimentation program, including the need for the right tools and expertise. Sadie discusses the growth of experimentation in businesses and the misconception that every business is doing it. She also highlights the value of failure and learning in experimentation, rather than focusing solely on winning experiments. Sadie compares her experiences working in an agency versus in-house, noting the differences in breadth of clients and control over experimentation processes. She also touches on the complexity of server-side testing and the buzz around personalization in the industry. In this conversation, Sadie and Alexandre discuss the topic of personalization in experimentation. They agree that personalization is often overhyped and that true one-to-one personalization is difficult to achieve. They share examples of how companies are using product recommendations and segmentation to personalize the user experience. They also discuss the importance of experimentation being customer-centric and based on research. They touch on the debate of centralized vs decentralized experimentation teams and the role of training in creating a culture of experimentation. They emphasize the importance of quality over quantity in experimentation and the need for clear goals and metrics to determine the success of an experiment. They also discuss the balance between velocity and quality in testing. They highlight the value of painted door tests and the importance of aligning experiments with the vision of the business. They conclude by defining experimentation as a means to validate and answer questions through rigorous testing.
Keywords
experimentation, CRO, stakeholder buy-in, challenges, tools, expertise, growth, failure, learning, agency, in-house, server-side testing, personalization, personalization, experimentation, product recommendations, segmentation, centralized vs decentralized, training, quality over quantity, goals and metrics, velocity, painted door tests, aligning with business vision
Takeaways
Stakeholder buy-in is crucial for starting an experimentation program
The right tools and expertise are necessary for successful experimentation
Failure and learning are valuable outcomes of experimentation
Working in an agency offers a breadth of clients, while in-house provides a deeper understanding of how large organizations operate
Server-side testing is more complex but allows for more advanced experiments
Personalization is often misunderstood and can be an extension of experimentation Personalization is often overhyped and true one-to-one personalization is difficult to achieve
Product recommendations and segmentation are effective ways to personalize the user experience
Experimentation should be customer-centric and based on research
There is a debate between centralized and decentralized experimentation teams
Training is important in creating a culture of experimentation
Quality is more important than quantity in experimentation
Clear goals and metrics are necessary to determine the success of an experiment
There is a balance between velocity and quality in testing
Painted door tests can provide valuable insights at a low cost
Experiments should be aligned with the vision of the business
Titles
The Journey into Experimentation
Agency vs In-House: Differences and Insights Quality Over Quantity in Experimentation
Setting Clear Goals and Metrics for Experiments
Sound Bites
"Stakeholder buy-in is crucial for starting an experimentation program"
"Failure and learning are valuable outcomes of experimentation"
"Working in an agency offers a breadth of clients, while in-house provides a deeper understanding of how large organizations operate"
"Personalization is definitely kind of a buzzword. I'll be excited when I see an article that won't have it."
"I've never seen a truly great UI personalization. I've never seen it before."
"Whether your experimentation team should be centralized or decentralized, there are pros and cons."
Chapters
00:00 The Journey into Experimentation
02:57 Challenges of Starting an Experimentation Program
06:06 The Value of Failure and Learning in Experimentation
09:55 Agency vs In-House: Differences and Insights
15:07 The Complexity and Potential of Server-Side Testing
33:35 The Hype and Reality of Personalization in Experimentation
38:08 Centralized vs Decentralized Experimentation Teams
43:30 The Role of Training in Creating a Culture of Experimentation
46:05 Quality Over Quantity in Experimentation
48:49 Defining Experimentation as a Means to Validate and Answer Questions
