Product experimenting

Product experimenting: which test should you choose?

Navigating the world of Product experimenting can sometimes be challenging. Which test should you choose? How many users should you have in it? How do you make sure your results are conclusive?

Part 1 of our data analyst’s guide to product experimentation: Choosing the right test for you. In this article, 3 different ways to perform product experiments will be discussed:

  • A/B testing
  • Multivariate testing
  • Multi-armed bandit testing

What is A/B testing?

A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.

A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.

Product experimenting

A/B testing is useful in 2 types of situations:

  • When you want to test 2 very different design directions against each other.
  • When one element of the page is up for debate.

A/B testing – Pros and Cons

ProsCons
– Simple to design and execute
– Statistically robust with smaller traffic
– Provides results that are easier for non-quantitative business teams to interpret and implement
– Limited to a single variable and a few variants of that variable
– Not possible to study the interaction between multiple variables within the same campaign

What is Multivariate testing?

The main difference between A/B and Multivariate testing is that the latter compares a higher number of variables. Hence, it gives us more information on how variables interact with one another. The purpose of Multivariate testing is to measure the impact of each design combination on the final goal.

Once a site has received enough traffic to run the test, the data from each variation is compared. This will then indicate which design combination is the most successful. Besides, it also reveals which elements are most responsible for the performance of a combination. This provides us with the possibility to see if there are any elements that jump out and have an extraordinarily significant impact.

Product experimenting

When to use Multivariate testing?

The most common use case for Multivariate testing is a page on which many elements are up for debate. Multivariate testing is only recommended for a site that has a substantial amount of daily traffic: the more variations that need to be tested, the longer it takes to produce meaningful data.

Multivariate testing – Pros and Cons

ProsCons
– Provides insights into interactions between multiple variables
– Helps identify which elements impact the performance
– Allow to compare many different versions of a page
– Requires more traffic than A/B testing
– Can become unmanageable as amount of combinations grows
– Can take more time to get up and running

What is Multi-armed bandit testing?

Multi-armed bandit is a type of testing that uses machine learning. It learns from data collected during the test and uses its new knowledge to dynamically allocate visitors. Its goal is to allocate users in such a way that better-performing variations continue to perform better and better. The flip side of this is that less-performing variations get less and less traffic allocation over time. This type of testing is particularly useful when you cannot afford to lose any conversion.

You should use Multi-armed bandit testing when: 

  • The opportunity/cost of loss is too high. For instance when selling diamonds or cars, a loss of conversion can cost thousands.
  • Optimizing click-through rates. For news outlets that cover time-sensitive events for example, the short shelf life of news pieces means that quick optimization is essential. 
  • Optimizing revenue with low traffic. If there is not enough traffic, A/B tests can take really long to produce statistical significance. In these cases, businesses may find it best to run Multi-armed bandit testing to be able to detect the potentially best version much earlier. This way, they can immediately capitalize on this discovery by directing an increasing amount of traffic to it.

A/B testing VS Multi-armed bandit testing

A/B testingMulti-armed bandit testing
– Goal is to collect data to make a critical business decision
– Goal is to see the impact of each variation with statistical confidence
– No need for interpretation of performance of variations – the goal here is to maximize conversions
– The window of opportunity for optimization is short-lived

We hope this product experimenting blog will help you along! Feel like reading more data stories? Then take a look at our blog page. Always want to stay up to date? Then be sure to follow us on LinkedIn.

Need some help?

2022-01-12MRF (1355)

Sophie Caro

“Data is een van de meest waardevolle bezittingen die je kunt hebben. Ik draag graag bij aan de groei en ontwikkeling van bedrijven door hen te helpen hun gegevens om te zetten in inzichten.”

More Data stories

google ads traffic
Data stories

Why GA4 classifies Google Ads traffic as (Organic) and how to fix it

If you rely on Google Analytics 4 (GA4) and noticed that some of your Google Ads traffic is showing up under the campaign name “(organic)”, you might be wondering why...
BLOG_koekje
Data stories

Five Ways to Enhance Your First-party Data Strategy

Google planned on phasing out third-party cookies due to issues mainly concerning privacy, at the end of 2024. However, they have postponed this phase-out once again, giving businesses (and Google)...
European Women in Technology
Data stories

European Women in Technology 2024 – Part 2

On May 26th and 27th, our team members Yvette and Sophie attended the 2024 installment of European Women In Technology. This event’s main purpose is to share ideas and discuss...
BLOG_ewit
Data stories

European Women in Technology 2024 - Part 1

On May 26th and 27th, our team members Yvette and Sophie attended the 2024 installment of European Women In Technology. This event’s main purpose is to share ideas and discuss...
Ontwerp zonder titel
Data stories

An algorithm for automatic data analysis, explained with a hint of orange

Over the past few months, The Data Story has had its first intern! Dorian is now finishing his Master’s in Data Science with a graduation project, also at The Data...
BLOG_afbw2
Data stories

Analytics for a Better World

For the second year in a row, our team-member Sophie Caro attended the Analytics for a Better World conference on May 14th. Nowadays, analytics play an important role in increasing...
nl_NLNederlands