What are the most important elements to prioritize when conducting A/B testing for your website or app?

If you are already familiar with the audience, it is good to start with some assumptions of what the results will be, based on the historical data. You might expect that one test will lead to a bigger impact than the other. Later on, you can confirm if this hypothesis is true or not.

As in any scientific experiment, you have to formule your hypothesis clearly. As the article states, it has to be specific, measurable, achievable, relevant and time-bound. The word "measurable" is key here, as it means that the hypothesis must be expressed quantitatively, thus the experiment can be replicated.

In my experience, before I conducted A/B testing for contact centres I documented and formulated a hypothesis that staff productivity will be increased by the removal of a test metric and this would be measured by an increase in daily answered calls. For our sample size, we selected a subset of employees from three departments that represented the test group (where we removed one KPI from their scorecard) and also selected a second subset of employees from two other departments who represented the control group (i.e. normal business conditions). The duration of the A/B test was one month as agreed by the project sponsors and data was collected and analyzed weekly to compare performance across 4 KPIs between the test and control groups.

Again, you are preparing a right experiment here. Segregating the audience to build a control group and a test group. The results for the test group will be compared against those for the control group, to determine is the hypothesis can be validated. Is a correlation is not clear, then the results are not conclusive.

If your hypothesis is measurable, then it can be expressed as a quantitative variable. The outcome of the test should also be expressed as a numeric variable. Then, by means of applying statistical methods, you can establish the correlation between both variables, which might help you support or reject the hypothesis.

For digital, direct-to-consumer propositions you need need to be mindful of the traffic to and engagement with your product / service / app. Unless you are a company that receives tens of thousands of engaged users each month, then you might not be able to A|B test many things simultaneously. This because you will not have enough sample size to test and evaluate. Prioritising what to test is key.