What is A/B Testing and Why Your Online Shop Should Be Doing It

Owning an online shop means constant change. Sometimes it’s just adding and removing products from your catalogue. Sometimes it means optimising the layout of your website. When it comes to the optimisation of your website, it might not always be very clear what the right decision is. That’s where A/B testing can really help you make the right call.

In today’s blog, we’ll talk about how you can plan an A/B test, what elements you can test, and other things to keep in mind.

Table of contents:

What is A/B testing?
Planning an A/B test
How to run an A/B test
What to test in an A/B test
Things to keep in mind
Tools for A/B testing

What is an A/B test?

An A/B test is pretty much what it sounds like...

A/B tests (a.k.a. split tests) are carried out in order to simultaneously test two or more variations of a (single) specific element of your business (e.g. the image used in your online ad). These variations will be displayed to (roughly) an equal amount of users and the data collected on those interactions are then analysed to determine a "winner".

The winning version will/should be permanently implemented once it is deemed the most effective variation.

Planning an A/B test

When testing what changes you can make to your business, there are two basic categories. You can test:

1. on-site elements of your business (your website)
2. off-site elements (advertisements, marketing emails).

The goals of these tests are the same: improving the effectiveness of your business to sell more.

Planning ahead of time is important. You’ll want to decide which elements you’ll be testing beforehand. Though this might sound obvious, you shouldn’t go overboard with testing every little element on your website. You’ll really want to hone in on those elements that lead to clicks and, more importantly, conversions.

With planning in mind, you should also remember that you should only test one thing at a time within a certain A/B test. For example, just test new product images before testing new product descriptions on a specific product page.

The reason for this is that you want to be able to clearly point to one factor when analysing your results, whether positive or negative. If you test, for example, new product images, CTAs, and product descriptions on a single product page at the same time, how will you know which elements had a strong impact and which ones had little to no impact?

The exception to this rule is probably ad campaigns. Because many ad campaigns are simple texts or a combination of texts and images, you could theoretically test all the different combinations simultaneously. For example, you could test 2 images and 2 headlines and mix and match them so each version is covered (i.e. 4 total variations).

Testing these 4 variations will give you a great overview of which combination worked the best. This way, you've essentially carried out 2 A/B tests at the same time.

It’s also important to think about what kinds of results you’ll be expecting to see. Make sure you know what results you’re currently getting (i.e. before running the test) in order to compare them to the results after you've finished the A/B test.

Some A/B tests assign “A” to be the current process, where “B” is the new element. Others will make both “A” and “B” new elements, and compare those to what is currently in place.

When testing multiple new elements, be sure that at least one of them actually outperforms the current element before replacing the original. Otherwise, it would be wise to keep the current elements in place. Ideally though, the winner will be obvious.

The basics of an A/B test: Two versions of the same website are displayed to equal amounts of people. In this case, the 25% conversion rate is clearly the winner!
(Source: Shutterstock/OneLineStock.com)

How to run an A/B test

Depending on what you want to test, you'll need to use different tools in order to test them.

For example, if you want to run an A/B test in your Instagram ads, that's easily done through the Facebook Ads Manager. Most ad platforms have A/B testing options built into the platform.

However, if you want to run A/B tests on your website, this is usually done easily through the shop software. Platforms like Shopify & WooCommerce have A/B testing tools built in, so setting up a test is super easy.

What to test in an A/B test

Wanting to optimise your website is great. Some changes can really bring dramatic improvements to your sales and that’s really what it’s all about, isn’t it? Before you make any changes (or test those changes), let’s look at some elements that you might want to consider testing.

You could test many of the following elements:

Headlines
CTA buttons
Website texts
Emails
Website design
Trust signals
Ad design
Ad targeting

Of course, there are plenty more elements you could test. Hopefully, this list inspires you to look at your website a little bit differently. For more details, let's explore those elements:

Headlines

If your company uses a content marketing strategy, headlines are what draw people in and inspire them to click. From an SEO perspective, this is really important as well. See what helps you rank higher in search engines, which will basically help you find out if more people are clicking on your ads, links, and posts.

Below, you can find an example of an A/B test I am running for banners that appear throughout the blog. I want to inspire my readers to download a calendar that has an overview of the major holidays throughout Europe. Using the same image, I only tweaked the headlines:

Be careful though. Click-bait headlines might lead to more clicks in the short-term, but if your content doesn't match your headlines, your bounce-rate will probably shoot up as well.

CTA buttons

CTA (Call-to-Action) buttons can be very significant. In the world of e-commerce, buttons are often used to add a product to the shopping cart. However, you might also use buttons to collect leads (e.g. newsletter sign-ups). There are many different uses, but generally speaking, they’re important for conversions in your business.

What they look like (images with texts vs simple buttons) and where they’re placed (at the top, middle, and/or bottom of your page) can also equate to big changes in performance.

Texts

Similarly to headlines, the copy of your product pages, landing pages, and emails can help you rank higher on Google. Good copy will also turn visitors into shoppers. Changing, tweaking, and testing your copy may well be worth A/B testing.

Emails

Speaking of emails, you can A/B test these as well. A huge factor for improving opening rates is the subject line. If you’re mailing list is long enough, consider sending out the same email with different subject lines to see which one is more effective (the same goes for newsletters).

Similarly, you may want to test the content of the email to see if certain language works better than others with your customers. CTA buttons are often found in emails as well.

Website design

This is, of course, a big one. Testing different elements of your website might show different results. On your homepage, you might want to change the header image and see if your bounce rates improve. You might want to change the layout or the menu buttons.

Shutterstock/Photographee.eu

If you open a website in a new country, you might want to test some different variations of your website to see how the new market reacts. Obviously, there is a lot you can do here, so choose the elements you think could have the biggest impact.

Trust signals

Taking website design a step further, you could also test how certain trust elements affect your conversion rates.

The most obvious example would be the placement of customer reviews throughout your website. Service reviews could be displayed on your homepages and throughout the checkout process, while product reviews would be prevalent on product pages and product category pages, for example.

Another trust element to test could be certifications. For example, the Trusted Shops Trustbadge displays both service reviews as well as the Trustmark, which is powered by a 30-day Buyer Protection.

In the example below, you can see how Kelvin Power Tools displays a link to their reviews in their navigation, but also displays their overall rating (as well as the Trustmark certification) in the lower-right corner of the homepage.

Lucardi actually performed an A/B test for the Trustbadge, which you can read about here: "Success story: Lucardi".

Ad design

As with most of the elements mentioned above, small changes can make a big difference sometimes. You might think that a certain image works really well in your Facebook ad, but you might well be surprised.

We once A/B tested an image of a shirtless man slumped on the couch playing with his tablet vs. a photo of me. Well, needless to say, I was surprised (and very disappointed) that Mr. Topless outperformed me (by a lot!). Besides images, you can also change the texts/headlines as well as the times that your ads are shown.

A/B testing revealed that a guy who kind of looks like me (left) led to more clicks than the real me (right)!... Not my best week. (shutterstock.com/Burdun Iliya)

Targeting

If you run an online shop, there’s a good chance you use Google or Facebook ads. They offer great targeting options to hone in on your target audience. If you’re a relatively new business owner, you might not know who your true audience is.

Be careful. You might think you know who your audience is, but you might be wrong about that. There are countless success stories of companies who ended up shifting their target audience to a different group and saw their sales skyrocket.

So, it goes without saying that testing out different audiences might surprise you. The info you get from testing these different ad variations can help you build a buyer persona, which can be helpful for future marketing efforts.

Recommended reading:
How to Use Micro-targeting in E-commerce

Things to keep in mind

Besides knowing what to test, there are some other things to keep in mind.

Give it time

The question of how long to run your A/B test is very common. It really comes down to having a significant result. If you’re running a high traffic site, you might be able to get some accurate numbers quite quickly (e.g. one week), but chances are that you’ll need to run these tests for a few weeks (e.g. 2-6 weeks).

There are some tools (more on that below) that can help you figure out how many people you need to run this test on before you can consider your results accurate.

Similarly, you must be careful not to let these tests run too long. Why? Sometimes, other variables can affect the outcome of your results. For example, I wouldn’t recommend running an A/B test during Christmas because shopping habits tend to be quite different during the holiday season (the exception, of course, would be if you specifically want to test some “Christmas elements” on your page).

Other variables that might affect your results over time include:

seasons (time of year)
holidays
ads you’re running during the test
press mentions of your business (positive or negative)

Beware of sample bias

When you’re running these experiments, you should have some kind of hypothesis as to the outcome. Naturally, you want to be right when it comes to predictions. Who doesn't enjoy being right,... right?

However, some marketers may unknowingly stop tests early after achieving the outcome they predicted. Again, it’s important to give your test enough time to be sure that it is accurate and tells a full story. Don't let your own predictions or your ego get in the way of getting accurate results.

Ending an A/B test too early is like an incomplete puzzle. You might think you know what the image is, but you can't be sure unless you really complete it.
(source: shutterstock.com/tadamichi)

Test multiple options

As mentioned above, some A/B tests will involve multiple variations for one element. This is fine, but only if you have enough traffic to make sure that your results are reliable.

Testing more than two variations is also known as A/B/C tests or A/B/n tests. Generally speaking, we don’t recommend testing multiple elements on one page at once, though some marketers don’t have a problem with this if it's done correctly.

Segmentation

Testing A versus B might seem pretty straightforward. You might think that “A” leads to more conversions than “B”, so “A” is clearly the winner. However, it’s important to look at the details of the results. Although “A” performed better overall, it might be the case that “B” significantly outperformed “A” on mobile devices. Keep this in mind when analysing your results. Some other factors to be aware of are:

Weekend visits vs weekday visits
Source of traffic (people who landed on the page from an ad might have different expectations from organic visitors)
Returning customers vs new customers
Desktop vs mobile
Time of day
Men vs women
Geographical locations
Age range

Tools for A/B testing

There are plenty of tools to help you with your tests. Here are some of the most popular tools:

A/B testing tools

A/B Testing Calculators

Conclusion

A/B testing can really help optimise your website and marketing efforts. Just remember to keep tests running long enough to know that your results are accurate, but not too long that they are skewed by other factors. Also, remember to store this information for future reference in case other stakeholders want to see this data.

All this information can really help your business improve its conversion rates. After all, if your website is your digital storefront, you’ll want to make sure you get as much “foot traffic” into your shop as possible (e.g. testing your ads). Once they are in your shop, you'll want to turn those window shoppers into loyal customers (e.g. test your "Buy now" buttons).