Product experimenting

Product experimenting: which test should you choose?

Navigating the world of Product experimenting can sometimes be challenging. Which test should you choose? How many users should you have in it? How do you make sure your results are conclusive?

Part 1 of our data analyst’s guide to product experimentation: Choosing the right test for you. In this article, 3 different ways to perform product experiments will be discussed:

  • A/B testing
  • Multivariate testing
  • Multi-armed bandit testing

What is A/B testing?

A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.

A/B testing is an experiment in which two or more variations of a page are shown to users at random. Statistical analysis is then used to determine which of them performs better. We measure their performance by considering one or multiple conversion goals, as well as whether a possible improvement is statistically significant. Examples of conversion goals are purchases, clicks and subscriptions.

Product experimenting

A/B testing is useful in 2 types of situations:

  • When you want to test 2 very different design directions against each other.
  • When one element of the page is up for debate.

A/B testing – Pros and Cons

ProsCons
– Simple to design and execute
– Statistically robust with smaller traffic
– Provides results that are easier for non-quantitative business teams to interpret and implement
– Limited to a single variable and a few variants of that variable
– Not possible to study the interaction between multiple variables within the same campaign

What is Multivariate testing?

The main difference between A/B and Multivariate testing is that the latter compares a higher number of variables. Hence, it gives us more information on how variables interact with one another. The purpose of Multivariate testing is to measure the impact of each design combination on the final goal.

Once a site has received enough traffic to run the test, the data from each variation is compared. This will then indicate which design combination is the most successful. Besides, it also reveals which elements are most responsible for the performance of a combination. This provides us with the possibility to see if there are any elements that jump out and have an extraordinarily significant impact.

Product experimenting

When to use Multivariate testing?

The most common use case for Multivariate testing is a page on which many elements are up for debate. Multivariate testing is only recommended for a site that has a substantial amount of daily traffic: the more variations that need to be tested, the longer it takes to produce meaningful data.

Multivariate testing – Pros and Cons

ProsCons
– Provides insights into interactions between multiple variables
– Helps identify which elements impact the performance
– Allow to compare many different versions of a page
– Requires more traffic than A/B testing
– Can become unmanageable as amount of combinations grows
– Can take more time to get up and running

What is Multi-armed bandit testing?

Multi-armed bandit is a type of testing that uses machine learning. It learns from data collected during the test and uses its new knowledge to dynamically allocate visitors. Its goal is to allocate users in such a way that better-performing variations continue to perform better and better. The flip side of this is that less-performing variations get less and less traffic allocation over time. This type of testing is particularly useful when you cannot afford to lose any conversion.

You should use Multi-armed bandit testing when: 

  • The opportunity/cost of loss is too high. For instance when selling diamonds or cars, a loss of conversion can cost thousands.
  • Optimizing click-through rates. For news outlets that cover time-sensitive events for example, the short shelf life of news pieces means that quick optimization is essential. 
  • Optimizing revenue with low traffic. If there is not enough traffic, A/B tests can take really long to produce statistical significance. In these cases, businesses may find it best to run Multi-armed bandit testing to be able to detect the potentially best version much earlier. This way, they can immediately capitalize on this discovery by directing an increasing amount of traffic to it.

A/B testing VS Multi-armed bandit testing

A/B testingMulti-armed bandit testing
– Goal is to collect data to make a critical business decision
– Goal is to see the impact of each variation with statistical confidence
– No need for interpretation of performance of variations – the goal here is to maximize conversions
– The window of opportunity for optimization is short-lived

We hope this product experimenting blog will help you along! Feel like reading more data stories? Then take a look at our blog page. Always want to stay up to date? Then be sure to follow us on LinkedIn.

Need some help?

2022-01-12MRF (1355)

Sophie Caro

“Data is one of the most valuable assets you can have. I love contributing to companies’ growth and development by helping them turn their data into business insights.”

More Data stories

BLOG_afbw2
Data stories

Analytics for a Better World

For the second year in a row, our team-member Sophie Caro attended the Analytics for a Better World conference on May 14th. Nowadays, analytics play an important role in increasing...
BLOGquality (1)
Data stories

Building High Performance Data & Analytics Teams

When we work together with our partners, we often assist them in building a data team. Numerous screenings and interviews all demonstrate one thing: finding the right candidate for the...
Data quality
Data stories

Data Quality is not difficult - how automation simplifies

When companies start collecting more data, data quality eventually becomes a topic of discussion. With more plants in your garden, maintenance is a larger responsibility. It is good to be...
clean ai
Data stories

AI needs clean, high-quality data - here’s why

With AI becoming more and more popular, its usage as a technology as well as a buzzword is growing ever so quickly. Even so, the title of this blog contains...
Product Owner
Data stories

Unlocking Data Potential: Role of a Product Owner in Data Teams

In the ever-evolving landscape of data science and analytics, organisations are recognising the need for a holistic and strategic approach to manage their data and turn it into value. One...
BLOGuit
Data stories

What is Server-side tagging in Google Tag Manager?

You’ve probably heard about Server-side tagging and might be wondering “What is it exactly?” and “How is it any different than the current Google Tag Manager setup?”. This blog will...
en_USEnglish