Experimentation-driven Product Development

Welcome to the Data Science Growth Series hosted by PrepVector! 👋 In this series, we help you up-level in your career by bringing the latest insights on data science by industry experts through events, articles, and webinars. I recently had the privilege of speaking with the data science students at the University of Connecticut about Experimentation-driven Product Development and the role of data scientists in building great products. Thank You Hari Patchigolla and Prof. Jeremy Teitelbaum for the opportunity!

Whether you are a seasoned professional, a student learning about experimentation for the first time, or simply curious about how we can build better products, we hope this newsletter will provide valuable insights and inspiration. So, let's dive into the world of experimentation!

About the Host:

Manisha Arora is a Data Scientist at Google with 10+ years of experience in driving business impact through data-driven decision-making. She is currently leading ad measurement and experimentation for Ads across Search, YouTube, Shopping, and Display. She works with top Google advertisers to support their marketing and business objectives through data insights, machine learning, and experimentation solutions.

Introduction:

Experimentation is a powerful tool for product growth. It is a way for businesses to test and validate new ideas, strategies, and tactics for driving growth.A/B testing involves comparing two versions of a product feature or design element to determine which one performs better. Product Managers use A/B tests to make data-driven decisions, optimize product features, and improve user engagement.

In this newsletter, we’ll dive deep into the history of experiments, the role of Product Managers and Data Scientists, real-world examples, and the future of experimentation.

Experimentation Overview:

The origin of experimentation is hard to trace but the first instances can be traced back to 1800s when the homeopathic medicine conducted randomized trials to test the medicines. However, the first significant leap was made when Fischer conducted experiments on crop cultivation that boosted crop production. Since then Fischer has published multiple scientific studies that introduced the concept of statistical design and experimental design.

Since then the concept of experimental design has been widely adopted across industries. The healthcare industry has been using the principle of experimentation to test drug efficacy. The tech industry has recently adopted experiments to improve their products iteratively.

What is Experimentation? 

Experimentation is the action or process of trying new ideas, methods, or activities, seeking innovation through controlled testing.

Rather than making decisions based on intuition, experiments help us get data to validate (or disprove) our assumptions. In its simplest terms, A/B testing allows Product Managers to make decisions with confidence by testing two versions of a feature or design element against each other on a small sample of users. The winner of the test is the feature we ship to the larger population.

And it all starts with a hypothesis! A hypothesis is a proposed explanation about a relationship between two observations made with limited evidence. It serves as a starting point for further investigation and fosters innovation.

A good hypothesis is:

• Testable
• Clearly-defined
• Data-driven
• Scalable

Ideas foster innovation. Experiments kill the opinion of the HiPPO (highest-paid person’s opinion) in the room and let the best ideas win, which leads to data-driven decision-making. Hence, it is crucial to establish a robust experiment design framework.

The Power of Analytics for Product Insights:

In recent years, analytics has evolved to offer targeted, specific, and actionable insights tailored for product growth and measurement. This specialization has given rise to the field of product analytics, where the focus is squarely on understanding user behavior, product usage, and the impact on business outcomes.

PM-DS Collaboration between Product Managers (PMs) and Data Scientists (DSs) is essential for creating and maintaining a robust experimental design. PMs provide domain knowledge and insights, while DSs bring analytical expertise to the table. Together, they work to design experiments that are both scientifically sound and aligned with business objectives.

Product analytics allows you to answer pivotal questions, such as:

1. What’s the difference in activity between people who churn and people who upgrade?

By comparing the actions of users who churn with those who upgrade, you can identify the key differentiators. This insight can help you tailor retention strategies to prevent churn and encourage more upgrades.

1. How engaged are people during the weekend vs weekdays, and what primary features drive that engagement?

Understanding when users are most engaged and which features drive that engagement can inform your marketing and product development strategies. For example, you can schedule feature releases or promotions during peak engagement times.

1. What factors drive users to return to our product within X days?

User retention is a critical metric. By pinpointing the factors that encourage users to return within a specific timeframe, you can design retention-focused campaigns and product improvements.

Data Scientists answer these questions by looking through website data to identify patterns and understand user behavior. These insights help identify opportunities for product iterations, which are then tested through experiments.

How Data Scientists Drive Experiments:

1) Design Experiment:

Based on the business goal, we can decide what type of experiment is best suited.

Incrementality Experiments: These experiments are typically run to assess the impact of a new feature or change on overall product health. Data Scientists measure the absolute performance difference before and after the change, helping the team understand the direct effect of the feature launch.

Optimization Experiments: These experiments focus on testing relative performance. They aim to find the best-performing variation of a feature by comparing multiple options. Data Scientists assess which variation has a positive impact on user engagement and business outcomes.

Causal Analysis: Similar to incrementality experiments, causal analysis experiments evaluate the impact of a feature launch on product health by assessing cause-and-effect relationships. These tests provide insights into how an intervention (like introducing a new feature) might affect key metrics.

2) Design Statistical Parameters

Data Scientists employ statistical concepts to ensure the validity and reliability of experiments. This includes determining errors, sample size, and duration:

• Type 1 Error (alpha): 

This represents the probability of erroneously rejecting the null hypothesis (H0) when it is, in fact, true. In other words, it is the likelihood of wrongly concluding that the test variation has a positive impact when it doesn't.

• Type 2 Error (beta): 

This is the probability of failing to reject the null hypothesis when it is false. It signifies the likelihood of wrongly concluding that the test variation does not have a positive impact when it actually does.

• Power of the Test (1-beta): 

This represents the probability of correctly rejecting the null hypothesis when it is false. In other words, it's the likelihood of correctly concluding that the test variation has a positive impact.

These statistical parameters, including the desired level of alpha and beta, help determine the sample size and the expected duration of the experiment. The trade-off between type 1 and type 2 errors plays a significant role in experimental design.

3) Analyze Results

At the end of the experiment, Data Scientists analyze the data to draw meaningful conclusions:

• Confidence Interval Estimation: 

Confidence intervals are used to determine the statistical significance of the experiment results. These intervals help ascertain the range within which the true effect of the experiment is likely to lie.

• Practical Significance: 

While statistical significance is essential, practical significance is equally important. It involves assessing whether the experiment's results make sense from a business perspective and whether the estimated effect would generate a meaningful impact.

• Metric Movements & Tradeoffs: 

In complex experiments, multiple metrics may move in different directions. For instance, an increase in visitors might be accompanied by decreased conversions. Data Scientists must determine which metrics are crucial for decision-making, the reasons behind their movements, and any expected trade-offs.

The Future of Experimentation:

These trends are shaping the data science of tomorrow. Here are some key aspects that are defining the future of experimentation.

• Cookieless Future: With the focus on user privacy and the end of third-party cookies, experimentation adapts to privacy-safe methods.

• Velocity and Accuracy Balance: Striking the right balance between speed and accuracy is a challenge. Companies are investing in ways to optimize testing processes without compromising quality.

• Collaboration is Key: Experimentation is a team effort involving Product Managers, Engineers, Designers, and Marketers. Tools and methods for seamless collaboration have been evolving and there will be an increased focus on this collaboration to ensure robust experiments leading to meaningful changes.

• Adapting to Regulations: Evolving data privacy regulations require quasi-experimentation to address challenges like user opt-in, ensuring accurate and compliant results.

It has been an absolute pleasure to chat with the bright minds of the University of Connecticut. If you are exploring Data Science roles and need more information, reach out to me.

If you are looking to hone your skills in Data Science, PrepVector offers a comprehensive course led by experienced professionals. You will gain skills in Product Sense, AB Testing, Machine Learning, and more through a series of live coaching sessions, industry mentors, and personalized career coaching sessions. In addition, you will also compound your skills by learning with like-minded professionals and sharing your learnings with the larger community along the way.

Book a free consultation to know more!

Kudos to Sujithra Gunasekar for helping draft this article. 👏 Subscribe to this newsletter to stay tuned for more such events!