How do you decide when to stop or iterate on an A/B test?

Para decidir parar ou iterar sobre um teste A/B eu analiso o seguinte: 1. Métrica de segurança: Se for ruim com alta confiança, mantenho o controle. Se for ruim com baixa confiança ou boa com alta confiança, avanço. 2. Métrica de sucesso: Caso 1: Se a métrica de segurança for ruim, mantenho o controle, a menos que a métrica de sucesso seja boa com alta confiança. Caso 2: Se a métrica de segurança for boa, repito o experimento, a menos que a métrica de sucesso seja boa com alta confiança. 3. Lift e confiança: Se o lift for alto ou moderado com alto poder, faço o rollout. Se o lift for moderado com baixo poder, repito o experimento.

To decide when to stop or iterate on an A/B test, consider statistical significance, practical significance, and the impact on key metrics. Define your hypothesis by identifying the problem you want to address and formulating a specific prediction about the expected impact of your solution. Then choose relevant metrics to measure this impact—these can be product usage, conversion rates, or any other relevant KPIs. Ensure that these metrics align with your broader business goals and are actionable.

My recommendation is to not be afraid to get specific with your hypothesis. Customer segmentation is key for most businesses. Are you interested in finding out about the conversion rate of a specific demographic or location? Don’t be afraid to rest that! Do you have messaging that you believe would resonate better with Gen Z over other groups? Test it out!

A proper hypothesis comes in pairs: the Null Hypothesis (H0) and the Alternative Hypothesis (H1). The Null Hypothesis usually infers that there is no effect or difference, while the Alternative Hypothesis states what you want to prove. For instance: H0: The medicine does not affect blood pressure. H1: The medicine lowers blood pressure. Specifically for A/B testing, I frame the hypothesis as an "if -> then" statement: If a person takes the drug, then their blood pressure will lower by X%. By comparing the two hypotheses, it is easier to quantify the impact of our variables and determine whether our findings are significant or merely the result of random chance.

Las hipótesis y las métricas también pueden ayudar a las empresas a ser más estratégicas en su enfoque de las pruebas A/B. Al comprender cómo los cambios en los elementos de diseño o contenido pueden afectar los objetivos de la empresa, las empresas pueden identificar oportunidades para mejorar la experiencia del usuario y el rendimiento de sus sitios web, aplicaciones o productos. Por ejemplo, una empresa podría utilizar pruebas A/B para probar diferentes diseños de página de destino con el objetivo de aumentar las conversiones. Al comparar los resultados de las pruebas, la empresa podría identificar el diseño que es más eficaz para atraer a los visitantes y conducirlos a la acción deseada.

Tout dépend du secteur d'activité. Dans le secteur de la presse écrite ou de l'e-commerce, nous avons souvent assez de volume pour que l'outil d'A/B test nous donne une fiabilité du résultat sans devoir le calculer. Si ce n'est pas le cas, cela dépend du secteur ; généralement, on peut s'arrêter à partir de 5 000 visiteurs uniques testés.

When determining the sample size and duration for an A/B test, consider factors such as the desired level of statistical significance, the expected effect size, and the variability of the measured metrics. Use statistical power calculations to determine an appropriate sample size that can detect meaningful differences. Consider running the test for at least one full business cycle to capture potential fluctuations over time. A longer duration may be necessary for products with low traffic or longer sales cycles.

If you're using the Bayesian method (vs. Frequentist), you shouldn't need to pre-select your sample size... There's more than one way to skin a cat.

Don't forget segmentation here either. When choosing your sample, if possible try to get a sample not just of random "users" but of the user personas you're trying to target with your test. If your sample isn't representative of the users you're trying to target, your results could be skewed to the preferences of a group you're not interested in. So if you want to see which ad, for instance, most resonates with people with rheumatoid arthritis, make sure that that's the group you're testing on.

Don't always be married to a duration(creative ad testing). Some platforms like Meta can give you a read pretty quickly. You don't want to be stuck running an underperforming ad that's losing money just for the sake of a duration plan. Flexibility is key.

When evaluating the results of an A/B test, it's essential to consider both statistical significance and practical significance. Statistical significance helps determine whether the observed differences are likely due to chance or represent a real effect. This is typically assessed using p-values or confidence intervals. Practical significance, on the other hand, focuses on whether the observed differences are large enough to be meaningful in a business context. It's crucial to interpret the results in light of both statistical and practical significance to make informed decisions about implementing changes based on the test findings.

Los resultados y el nivel de confianza son esenciales para las pruebas A/B, ya que determinan si una variación es significativamente mejor que la otra. Para determinar si una variación es significativamente mejor que la otra, se suele utilizar un nivel de confianza del 95%. Si el tamaño del efecto de una variación es lo suficientemente grande y el nivel de confianza es lo suficientemente alto, entonces la variación es significativamente mejor que la otra. En resumen, los resultados y el nivel de confianza son factores importantes que deben considerarse al analizar los resultados de una prueba A/B. Al comprender estos factores, las empresas pueden tomar decisiones informadas sobre si implementar una variación o seguir probando.

Evaluate Uplift: Assess if the uplift is significant enough to justify a business case. Consider collecting more data if the uplift is small. Consider Sample Dilution: Be aware that the likelihood of significant results decreases over time, especially if early results are inconclusive. Manage Overlapping Experiments: If extending the test prevents launching new experiments, consider prioritizing new tests. Recognize Test Endpoint: Understand that continuing an experiment indefinitely is not productive; decide based on the current results and confidence levels. These considerations help determine whether to conclude the test or continue with further iterations.

- Consider the statistical tests you will use. You can perform simulations on your historical data to understand if the specified type I error is maintained and if the specified power is achieved. - To increase the sensitivity of the test, you can use dimensionality reduction methods. Start by filtering out outliers. After that, you can try using stratification and CUPED (Controlled-experiment Using Pre-Experiment Data). For ratio metrics, you can also use a method called linearization to convert it to a per-user format, after which you can apply the dimensionality reduction methods.

One important factor I saw missing in the other answers is the MDE, the minimum detectable effect. Quite often, the sample size is limited. Confidence level is set to a standard 95% and you'll want to go for a Power level of 80%. The MDE on the other hand then tells you how large of a change you are able to detect. This should heavily guide your A/B Testing strategy as it might turn out that there is a section in the app for which you can only measure huge changes with confidence. You should initially avoid those, since you might miss out on a lot of small wins. You won't notice them after all because your MDE is too high.

In many A/B test the challenger is too similar, an image or wording is slightly different, and thats why the results are underwhelming or unclear. I recommend testing radically different approaches - for example linking directly to a product detail page instead of a compare page. And test to impact the metric that matter: for e-commerce test to impact revenue or add-to-charts instead of CTR.

When determining the sample size and duration for an A/B test, consider factors such as the desired level of statistical power, the magnitude of effect you want to detect, and any constraints related to cost or time. It's important to strike a balance between obtaining a sufficiently large sample size to detect meaningful differences and avoiding unnecessary costs or delays. Additionally, considering trade-offs such as the potential impact on user experience or the risk of longer testing durations can help make informed decisions about the sample size and duration for an A/B test.

Además de los resultados y el nivel de confianza, hay otros factores y compensaciones que pueden influir en la decisión de detener o iterar en una prueba A/B. Estos factores incluyen: La importancia práctica: Es la relevancia y el valor del tamaño del efecto para los usuarios y los objetivos. El impacto comercial: El impacto comercial es el ingreso estimado o el ahorro de costos que la prueba A/B puede generar o ahorrar. El costo de oportunidad: El costo de oportunidad es el beneficio potencial que se pierde al no probar otra cosa. El riesgo: El riesgo es el daño potencial que la prueba A/B puede causar a los usuarios o la marca.

If you're testing UX flows, for instance, you'll want to make sure you're checking in with engineering to see if there's a significantly higher implementation cost to either of the tested scenarios. I've run A/B tests where the test was so conclusive in one direction that we ended up going with something that had a much higher implementation cost, but it's also been the case where the A/B test showed a slight preference for a solution but the implementation cost was so much more that we ended up going with the somewhat less effective solution.

Consider these factors and trade-offs: Statistical Significance: Monitor if results reach statistical significance within the expected timeframe. Test Duration: Longer durations may lead to sample pollution; stop if the test exceeds the planned period without clear results. Resource Allocation: Assess resource consumption versus insights gained. Redirect resources to other tests if more beneficial. Opportunity Cost: Evaluate other potential experiments that could be performed with the same resources. Business Impact: Consider the test's impact on key performance indicators and business outcomes. Balancing these factors helps in making informed decisions about the continuation or conclusion of A/B tests.

A/B testing is a critical way to make decisions based on reliable data directly from your customer base. But sometimes, there comes a point when too much data is available. That's when we can fall into the 'analysis paralysis' trap, or the inability to commit to a decision because of overthinking. Yes, testing in invaluable. But at a certain point, don’t neglect your gut instinct. Your perception of the right thing to do cannot always be replaced by testing. “Let’s just test it” can be a trap - be confident in the perspective you've developed based on data, experiences, trends, and what you believe to be best for your customers.

Deciding when to stop or iterate on an A/B test is pivotal in technology management. In my role as a CTO, I have found that it’s essential to focus on the insights and feedback from these tests. This data tells you not only if the test met its objectives but also what underlying factors contributed to its success or failure. Utilizing both quantitative data and qualitative feedback from users helps in making informed decisions about the next steps. Whether to refine the feature further or to roll it out broadly should be based on a comprehensive analysis of test results, aligned with overall business goals and user satisfaction.

When deciding whether to stop or iterate on an A/B test, it's crucial to learn from insights and feedback. Gather and analyze data to understand the implications of the test results. Consider factors such as statistical significance, practical significance, user feedback, and other relevant metrics to determine if there are actionable insights that can be gleaned from the test. If significant findings emerge or valuable feedback is received, it may be time to iterate based on these insights. However, if the results do not provide clear direction or are inconclusive, it might be necessary to stop the test and re-evaluate the strategy before proceeding further.

El aprendizaje de las pruebas A/B puede ser un proceso continuo. Las empresas deben estar dispuestas a revisar sus conclusiones y recomendaciones a medida que obtienen más información. También es importante compartir los conocimientos y comentarios de las pruebas A/B con toda la organización. Esto ayudará a garantizar que todos estén en la misma página y que las mejores prácticas se compartan y se apliquen en toda la empresa. En resumen, el aprendizaje es una parte esencial de las pruebas A/B. Al aprender de sus pruebas, las empresas pueden mejorar sus productos, servicios y experiencias de usuario.

Sharing the insights is one of the most valuable parts of the process and shouldn't be forgotten. But make sure you do it as a team. Present the results and discuss it. Poke holes in it as a team. Even when a test fails there's usually something to learn and feed into your next test, and that is where the compounding effect of both learning and performance takes place. Let your team rally around the data, bring alignment to your decisions and share the wins!

AB testing is a decision tool. By comparing A vs. B, we learn the impact of B and assume that it will generalize to the full population. What is often unclear is that AB tests come with some uncertainty: results can lead to false positives or false negatives. If you test a new conversion funnel and observe a statistically significant increase in CTR, there is some risk that it is a false positive (a random positive result). The opposite can happen when we fail to detect a positive impact (a false negative). This case often occurs when the studies are underpowered, and there are not enough users to draw sound conclusions. Discussing these risks upfront often leads to better planning and improved iterations in case of failure.

When deciding whether to stop or iterate, keep in mind the broader implications for your product or service. Consider how the changes align with your long-term strategy and user needs. Additionally, review any technical issues or limitations encountered during the test, which might affect your decision. For instance, if a test reveals significant issues with user experience, addressing these issues before making final decisions is crucial. Staying agile and responsive to both data and user feedback ensures a well-rounded approach to optimizing your A/B testing strategy.