Navigating the world of Product experimenting can sometimes be challenging. Which test should you choose? How many users should you have in it? How do you make sure your results are conclusive?
Part 1 of our data analyst’s guide to product experimentation: Choosing the right test for you. In this article, 3 different ways to perform product experiments will be discussed:
- A/B testing
- Multivariate testing
- Multi-armed bandit testing
What is A/B testing?
A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.
A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.
A/B testing is useful in 2 types of situations:
- When you want to test 2 very different design directions against each other.
- When one element of the page is up for debate.
A/B testing – Pros and Cons
| Pros | Cons |
| – Simple to design and execute – Statistically robust with smaller traffic – Provides results that are easier for non-quantitative business teams to interpret and implement | – Limited to a single variable and a few variants of that variable – Not possible to study the interaction between multiple variables within the same campaign |
What is Multivariate testing?
The main difference between A/B and Multivariate testing is that the latter compares a higher number of variables. Hence, it gives us more information on how variables interact with one another. The purpose of Multivariate testing is to measure the impact of each design combination on the final goal.
Once a site has received enough traffic to run the test, the data from each variation is compared. This will then indicate which design combination is the most successful. Besides, it also reveals which elements are most responsible for the performance of a combination. This provides us with the possibility to see if there are any elements that jump out and have an extraordinarily significant impact.
When to use Multivariate testing?
The most common use case for Multivariate testing is a page on which many elements are up for debate. Multivariate testing is only recommended for a site that has a substantial amount of daily traffic: the more variations that need to be tested, the longer it takes to produce meaningful data.
Multivariate testing – Pros and Cons
| Pros | Cons |
| – Provides insights into interactions between multiple variables – Helps identify which elements impact the performance – Allow to compare many different versions of a page | – Requires more traffic than A/B testing – Can become unmanageable as amount of combinations grows – Can take more time to get up and running |
What is Multi-armed bandit testing?
Multi-armed bandit is a type of testing that uses machine learning. It learns from data collected during the test and uses its new knowledge to dynamically allocate visitors. Its goal is to allocate users in such a way that better-performing variations continue to perform better and better. The flip side of this is that less-performing variations get less and less traffic allocation over time. This type of testing is particularly useful when you cannot afford to lose any conversion.
You should use Multi-armed bandit testing when:
- The opportunity/cost of loss is too high. For instance when selling diamonds or cars, a loss of conversion can cost thousands.
- Optimizing click-through rates. For news outlets that cover time-sensitive events for example, the short shelf life of news pieces means that quick optimization is essential.
- Optimizing revenue with low traffic. If there is not enough traffic, A/B tests can take really long to produce statistical significance. In these cases, businesses may find it best to run Multi-armed bandit testing to be able to detect the potentially best version much earlier. This way, they can immediately capitalize on this discovery by directing an increasing amount of traffic to it.
A/B testing VS Multi-armed bandit testing
| A/B testing | Multi-armed bandit testing |
| – Goal is to collect data to make a critical business decision – Goal is to see the impact of each variation with statistical confidence | – No need for interpretation of performance of variations – the goal here is to maximize conversions – The window of opportunity for optimization is short-lived |
We hope this product experimenting blog will help you along! Feel like reading more data stories? Then take a look at our blog page. Always want to stay up to date? Then be sure to follow us on LinkedIn.
English 
Nederlands