Call us at +44 20 33 64 59 06

Schedule a


What is A/B Testing and Why Your Online Shop Should Be Doing It

18.11.2020, 9m

Owning an online shop means constant change. Sometimes it’s just adding and removing products from your catalogue. Sometimes it means optimising the layout of your website. When it comes to the optimisation of your website, it might not always be very clear what the right decision is. That’s where A/B testing can really help you make the right call.

A/B testing is pretty much what it sounds like.

A/B testing is the process of testing out two or more variations of a specific element of your business (e.g. the images used in your online ads) simultaneously. These variations will be displayed to (roughly) an equal amount of users and the data collected on those interactions are then analysed to determine a "winner". 

The winning version will then be implemented throughout your website once it is deemed the most effective variation. In today’s blog, we’ll talk about how you can plan an A/B test, what elements you can test, and other things to keep in mind.

Planning an A/B test

When testing what changes you can make to your business, there are two basic categories. You can test:

    1. on-site elements of your business (your website)
    2. off-site elements (advertisements, marketing emails).

The goals of these tests are the same: improving the effectiveness of your business to sell more.

Planning ahead of time is important. You’ll want to decide which elements you’ll be testing beforehand. Though this might sound obvious, you shouldn’t go overboard with testing every little element on your website. You’ll really want to hone in on those elements that lead to clicks and, more importantly, conversions.

It’s also important to think about what kinds of results you’ll be expecting to see. Make sure you know what results you’re currently getting in order to compare them to the results after you've finished the A/B test.

Some A/B tests assign “A” to be the current process, where “B” is the new element. Others will make both “A” and “B” new elements, and compare those to what is currently in place. 

When testing multiple new elements, be sure that at least one of them actually outperforms the current element before replacing the original. Otherwise, it would be wise to keep the current elements in place. Ideally though, the winner will be obvious. 

graphic for ab tests
The basics of an A/B test: Two versions of the same website are displayed to equal amounts of people.  In this case, the 25% conversion rate is clearly the winner!

What to test in an A/B test

Wanting to optimise your website is great. Some changes can really bring dramatic improvements to your sales and that’s really what it’s all about, isn’t it? Before you make any changes (or test those changes), let’s look at some elements that you might want to consider testing.

You could test many of the following elements:

  • Headlines
  • CTA buttons
  • Website texts
  • Emails
  • Website design
  • Ad design
  • Ad targeting

Of course, there are plenty more elements you could test. Hopefully, this list inspires you to look at your website a little bit differently. For more details, let's explore those elements: 


If your company uses a content marketing strategy, headlines are what draw people in and inspire them to click. From an SEO perspective, this is really important as well. See what helps you rank higher in search engines, which will basically help you find out if more people are clicking on your ads, links, and posts.

Be careful though. Click-bait headlines might lead to more clicks in the shor-term, but if your content doesn't match your headlines, your bounce-rate will probably shoot up as well. 

CTA buttons

CTA (Call-to-Action) buttons can be very significant. In the world of e-commerce, buttons are often used to add a product to the shopping cart. However, you might also use buttons to collect leads (e.g. newsletter sign-ups). There are many different uses, but generally speaking, they’re important for conversions in your business.

What they look like (images with texts vs simple buttons) and where they’re placed (at the top, middle, and/or bottom of your page) can also equate to big changes in performance.


Similarly to headlines, the copy of your product pages, landing pages, and emails can help you rank higher on Google. Good copy will also turn visitors into shoppers. Changing, tweaking, and testing your copy may well be worth A/B testing.

keywords to improve SEO SEA whitepaper


Speaking of emails, you can A/B test these as well. A huge factor for improving opening rates is the subject line. If you’re mailing list is long enough, consider sending out the same email with different subject lines to see which one is more effective (the same goes for newsletters).

Similarly, you may want to test the content of the email to see if certain language works better than others with your customers. CTA buttons are often found in emails as well.

Website design

This is, of course, a big one. Testing different elements of your website might show different results. On your homepage, you might want to change the header image and see if your bounce rates improve. You might want to change the layout or the menu buttons.

different shades of color


If you open a website in a new country, you might want to test some different variations of your website to see how the new market reacts. Obviously, there is a lot you can do here, so choose the elements you think could have the biggest impact.

Ad design

 As with most of the elements mentioned above, small changes can make a big difference sometimes. You might think that a certain image works really well in your Facebook ad, but you might well be surprised.

We once A/B tested an image of me vs. a shirtless man slumped on the couch, playing with his tablet. Well, needless to say, I was surprised (and very disappointed) that Mr. Topless outperformed me (by a lot!). Besides images, you can also change the text and times that your ads are shown.

two similar looking models

A/B testing revealed that a guy who kind of looks like me (left) leads to more clicks than the real me (right)!... Not my best week. 


If you run an online shop, there’s a good chance you use Google or Facebook ads. They offer great targeting options to hone in on your target audience. If you’re a relatively new business owner, you might not know who your true audience is.

Be careful. You might think you know who your audience is, but you might be wrong about that. There are countless success stories of companies who ended up shifting their target audience to a different group and saw their sales skyrocket. 

So, it goes without saying that testing out different audiences might surprise you. The info you get from testing these different ad variations can help you build a buyer persona, which can be helpful for future marketing efforts. 

Recommended reading:
How to Use Micro-targeting in E-commerce

Things to keep in mind

Besides knowing what to test, there are some other things to keep in mind.

Give it time

The question of how long to run your A/B test is very common. It really comes down to having a significant result. If you’re running a high traffic site, you might be able to get some accurate numbers quite quickly (e.g. one week), but chances are that you’ll need to run these tests for a few weeks (e.g. 2-6 weeks).

There are some tools (more on that below) that can help you figure out how many people you need to run this test on before you can consider your results accurate.

Similarly, you must be careful not to let these tests run too long. Why? Sometimes, other variables can affect the outcome of your results. For example, I wouldn’t recommend running an A/B test during Christmas because shopping habits tend to be quite different during the holiday season (the exception, of course, would be if you specifically want to test some “Christmas elements” on your page).

Other variables that might affect your results over time include:

  • seasons (time of year)

  • holidays

  • ads you’re running during the test

  • press mentions of your business (positive or negative)

Beware of sample bias

When you’re running these experiments, you should have some kind of hypothesis as to the outcome. Naturally, you want to be right when it comes to predictions. However, some marketers may unknowingly stop tests early after achieving the outcome they predicted. Again, it’s important to give your test enough time to be sure that it is accurate and tells a full story.

lightbulb in a jigsaw puzzle

Ending an A/B test too early is like an incomplete puzzle. You might think you know what the image is, but you can't be sure unless you really complete it.

Test multiple options

As mentioned above, some A/B tests will involve multiple variations for one element. This is fine, but only if you have enough traffic to make sure that your results are reliable.

Testing more than two variations is also known as A/B/C tests or A/B/n tests. Generally speaking, we don’t recommend testing multiple elements on one page at once, though some marketers don’t have a problem with this if it's done correctly.


Testing A versus B might seem pretty straightforward. You might think that “A” leads to more conversions than “B”, so “A” is clearly the winner. However, it’s important to look at the details of the results. Though “A” performed better overall, it might be the case that “B” significantly outperformed “A” on mobile devices. Keep this in mind when analysing your results. Some other factors to be aware of are:

  • Weekend visits vs weekday visits

  • Source of traffic (people who landed on the page from an ad might have different expectations from organic visitors)

  • Returning customers vs new customers

  • Desktop vs mobile

  • Time of day

  • Men vs women

  • Geographical locations

  • Age range

Tools for A/B testing

There are plenty of tools to help you with your tests. Here are some of the most popular tools:

A/B testing tools

A/B Testing Calculators


A/B testing can really help optimise your website and marketing efforts. Just remember to keep tests running long enough to know that your results are accurate, but not too long that they are skewed by other factors. Also, remember to store this information for future reference in case other stakeholders want to see this data.

All this information can really help your business improve its conversion rates. After all, if your website is your digital storefront, you’ll want to make sure you get as much “foot traffic” into your shop as possible (e.g. testing your ads). Once they are in your shop, you'll want to turn those window shoppers into loyal customers (e.g. test your "Buy now" buttons).

checkout optimisation for cart abandonment whitepaper