A-B Testing What you Need to Know

  • Introduction

  • Why Should you Conduct an A/B Test?

  • A/B Testing vs. Multivariate Testing

  • A/B Testing Best Practices

  • What Variables Should you Test?

  • Running and Evaluating Tests

  • Conclusion


You can A/B test smaller, less significant elements right through larger and much more significant elements. Either way, the desired outcome of an A/B test is to ensure you understand what works best and what your audience enjoy the most. If you are aware of what they enjoy the most, you can exploit this and continue following the pattern of the best performing elements to better your business.


A/B testing enables you to experiment and see what option performs the best


A/B testing is important as it will, essentially, bring your business higher levels in most areas. It can cause higher conversion rates, higher engagement levels, and so on. When you’re conducting an A/B test your main aims should be to learn what will gain you all these higher results - so you can implement them.

Conducting an A/B test isn’t as easy as it seems; you have to be prepared to learn a lot before you begin!

Why Should you Conduct an A/B Test?

As determined, A/B testing means you’re testing what works best. What you test can range from copy, images, a whole page, and so on - allowing you to see which performs the best in terms of whichever metric you decide to measure.

So, why should you conduct an A/B test? They’re one of the easiest ways of determining what performs the best. Without conducting an A/B test, you’re limiting yourself to the experimenting you can do.


An A/B test allows you to change whatever you like about a certain thing, and see which one performs best


It gives you room to grow and prosper as a business; a chance that is available to everyone but enhanced when you conduct A/B tests as you’re making data-driven decisions, decisions that you know will work and be effective. 

A/B testing is, essentially, optimising your site, content, emails, etc. to be the best they can be. Without an A/B test, you can experiment - but it may take longer to fully comprehend what works and what doesn’t. A/B tests tell you immediately what performs the best.


A/B Testing vs. Multivariate Testing

It may be widely believed that A/B testing and multivariate testing are the exact same things, however, this is not true. They are similar, allowing you to do similar activities, but not the same.


A/B Testing

A/B testing only allows you to test one variable at a time. It, therefore, enables you to see which variable performs better and people enjoy the most.


Multivariate Testing

Multivariate testing differentiates from A/B testing due to it allowing you to test multiple variables at once. Essentially, multivariate testing is a series of A/B tests. 

Multivariate testing allows you to analyse how the little changes make a difference. They enable you to see what colour button for a CTA works best, what wording works best, and so on. All these smaller elements combine to make one big element (a web page, newsletter, etc.) - which can be tested with A/B testing. An overall conclusion for both A/B and multivariate tests would be knowing what to optimise and improve to increase a certain metric.


A/B Testing Best Practices

Only conduct one test at a time

Testing too many elements at once can cause confusion. When A/B testing, you want the most accuracy you can get. To get this accuracy, it’s best to conduct only one test on the same thing at once. Testing one hypothesis at a time allows you to really focus on the results and coming to much stronger conclusions than you would if you were testing more than one.

For example, if you want to segment your audience and test which landing pages works best; you’re not going to get accurate results for both if you test them simultaneously.


Test one variable at a time

As mentioned above, testing two things at once complicates results. It’s important to remember that this goes for testing variables too. Variables can include a CTA, images, copy, and so on. Testing one of these at a time enables you to pinpoint what changes made improvements, and come to another strong conclusion on what works best.


Test minor changes

It’s crucial to test minor changes. Every single element on everything you ever do is there for a reason, so why shouldn’t you test them and decipher what works best for the smaller elements?


 Or, test the entire element

A/B testing enables you to test elements on a much larger scale. This means you’re able to test the entire element. It’s good practice to design two completely different versions of what you’re wanting to test. It’s then a matter of measuring the results and necessary metrics and seeing which version performed better.


Measure everything that you can measure

Obviously, not every single metric is going to be relevant to your business. But it’s always good to measure everything that you can possibly measure. If you fail to measure important metrics, there’s a chance you won’t come to a correct conclusion and the results will be wrong.


Set up control and treatment

Setting up control and treatment ‘pages’ means having a set of pages, one being unaltered and one being the altered page. Have one page that isn’t changed or a page off your original website that you’ve made absolutely no changes to. Treatment pages are the pages that you’ve made explicit changes to, based on your control page/s.


Decide what you want to test

The tests you can carry out are endless. As mentioned a few times, it’s good practice to only test one variable at a time for a particular element. Due to the limitless amount of tests, it’s important you decide, as a team, what variables and elements you want to test and slowly work your way through them. This way, you’re much more organised and efficient. 


Split your sample group randomly

To make it fair on yourself, your audiences need to be split randomly rather than systematically. Splitting them up systematically only means each audience isn’t fair and equal. It’s important to have a fairly and equally split audience to get the most accurate results. Segmenting your audience means you automatically understand what they prefer; whereas a randomly split audience has a wide variety of personalities so every individual will enjoy different elements.


Test at the same time

This seems simple, but some may forget it. A/B testing needs to happen at the same time. Even if test A is ready a long while before test B, you must test them at the same time. Testing them at different times creates variables that you can’t change and that might make a huge difference in results.


Decide on significance of results before testing

You can’t just assume an element is better and brings in better results if it got one more click than the original element. You need to decipher, as a team, what result will be significant enough. If a result hits this significance, it will signify to you that it’s worth changing; and if it doesn’t, then perhaps wait longer before deciding what’s best for that element.


What Variables Should you Test?

There’s always endless amounts of variables you can actually test to see which perform better. However, some variables may not prove worth it. You want to test variables that will give you beneficial results; ones that will improve metrics such as click through rate. Try to focus on elements that will bring you results.


Elements you can optimise on a landing page:


1. Offers

Testing what offers work best is a good place to start. This is, essentially, testing what offers visitors like the most. Testing what offers give you the most conversions enables you to see what your audience like best and what gets people most interested.

Examples: eBooks, webinars, discounts, coupons, sales, demos, etc.


2. Copy

Without testing your copy, you’re not going to know which is more attractive and engaging to your audience. Try to consider every variable of copy you can test, whether it be a different tone, message, layout, and so on. Make sure your copy is easy to read and understand, no matter what.


3. Forms

Businesses use forms in a multitude of different ways. They are a great way to gather information and qualify audiences. You can A/B test forms by asking for different information and seeing which form performs best; or putting content behind a form and determining the percentage of visitors willing to fill a form in for content you’re offering.


4. Whole page 

While it may seem easier testing much smaller elements, to get the best results it’s good to test the whole page. Testing the whole page gives you much more substantial results, and allows you to come to an all round stronger conclusion. Drastically changing the whole page enables you to see what your audience prefers.


Elements you can optimise on a CTA button:


1. Placement 

No one knows where the best average place to put a CTA button is. That’s why it’s important to test out different placements of yours and see what placement gets the most clicks. It may take time to decipher what placement is best for your CTAs, especially as the optimised placement is most likely going to be different for each page or piece of content. Remember, everyones placement of CTAs is going to be different - you just have to see what works best for you.


2. Size

How big do you make your button? How small? If you’re unsure, test it! Test whether the size makes a difference and which size is best for the most clicks. Remember, you need a button that’s big enough to help draw attention but not too big where they become overpowering.


3. Colour

Much like every other aspect of a CTA, no one knows the correct answer to what colour is best for an all round optimised CTA button. It varies for every single business and that’s why it’s important to test out colours and see what colour/s bring the most attention and clicks. Try to choose a colour that makes sense for your brand to use it, something that fits in with the rest of the page - but also stands out.


4. Copy

There are hundreds of different statements that can go into a CTA. This is where A/B testing comes into action. You must test out what copy works best and gets your audience most engaged - resulting in the most clicks. An overall rule, though, is to keep your copy short and sweet; get straight to the point and tell your audience what you want them to do.


Elements you can optimise in email:


The elements mentioned above in terms of landing pages and CTA buttons can also be tested within emails, as well as everything below.


1. Format

Are you aware of the best formatting for your email? If not, it’s worth testing what layout drives the most engagement and interest. Layouts can vary anyway depending on what type of email you’re sending - so it’s a good idea to test all different layouts for all different types of emails.


2. Timing

The best time to send out emails will completely vary depending on your business and audience. A/B testing enables you to see what time is best to send out emails - more specifically, what time gets you the most opens and engagements.


3. Sender

Often, people will look at the email sender and decide whether they trust the email. If it’s sent from an email address that looks impersonal, suspicious, unprofessional, and so on - they’re more than likely going to delete it, mark it as spam, etc. It’s always a good idea to A/B test the email address you send your emails from. Personal emails add a human side to your business, whilst business addresses show you’re professional and can be trusted.


4. Subject lines

Whilst the subject line is going to be different for each email, testing subject lines enables you to make decisions on what tone, format, style, etc. of subject line works the best and reiterate your findings throughout the majority of your emails. A subject line is the most significant factor for luring your audience into opening and reading the email, so you have to get it right.


Running and Evaluating Tests

Conducting a landing page test

Landing page A/B testing gives you just one URL, with two or more versions of that specific page. People wanting to visit that URL will then be randomly delivered to one of the versions. Your analytics tool with then have the job of measuring the results from every page and showing you all the metrics. From these metrics you can then derive a conclusion and understand what page worked best, and more specifically what worked best on the page. It’s very important to look at all the relevant metrics and decide what pages performed the way you wanted.


Conducting a CTA test

This is very similar to landing page tests, in the fact that people visiting your website/a web page will randomly be delivered to the pages with different CTAs on. Whilst a landing page test has so many metrics you can measure and conclude on, a CTA test is pretty much about how many clicks the different CTAs get.

However, it’s important to remember that when you conduct a CTA test - make sure it’s the only test you’re conducting on that particular page. If you A/B test a CTA as well as the whole page, it won’t give you an accurate representation of what performed well. You need full concentration on the CTA, to understand the results and recognise what performed the best overall.


Conducting an email test 

Some email providers will make A/B testing an email easy for you and do most of the work - for example split your subscribers into random lists and send every different email out to one of those lists.

If you have to do it manually, all you have to do is split your subscriber list equally and fairly, so the types of audience is varied in each list. You then must write two or more emails that differ and send them out to your lists to see which email performs the best. Similar to the landing page test results, there are a multitude of metrics you can keep an eye on for emails but it’s important to decide what metrics are most important to your business and see which email performs best for those metrics



A/B testing is the easiest way to optimise elements such as conversion rate, engagement levels, click through rate and so on.


A/B testing will give you the information needed to increase all these metrics and further improve your business


You can A/B test small and bigger elements depending on what your goals are. If you’re wanting to increase conversion rate, it’s best to A/B test aspects on a bigger scale, for example a landing page. However, if you’re just starting out and wanting to increase clicks, it can be a good idea to begin with conducting an A/B test on the CTAs you use. 

Whatever you A/B test, there needs to be an overall reason for it. Complete your A/B test with goals in mind, with some metrics to reach for - rather than just A/B testing elements for the sake of it. It’s important to constantly improve your business performance in a multitude of areas, and A/B testing can assist this continuous improvement.