If you want to know which design is best for your website or what text makes your emails more enticing, guess work is not going to cut it. For the best results you need to tap into your greatest resource: your customers.

A/B testing is a brilliant tool that allows you to gather data from your customers and make better decisions based on their feedback.

Remember Barack Obama’s presidential campaign in 2008? It appeared to be innovative in many ways but few know that, back in 2007, this team acquired a new member: Dan Siroker. He was a Google product manager at that time and as a digital advisor for the presidential campaign, he did what he could do best – applied his experience and showed Obama’s team how to do A/B testing.

Using that tool, Siroker and other specialists changed the campaign website. The improvements resulted in 40% more signups and $75 million in money raised.

Pretty impressive huh? If you want to start seeing improved results for your business, read on to find out what A/B testing is and how to use it.

  • What Is A/B Testing? 🤷‍♀️
  • Types of A/B Testing 🕵️‍♀️
  • Why Is A/B Testing Important for Your Web Product? 💸
  • A/B Testing Guide 📝
  • How to Conduct A/B Testing 👩‍🏫
  • Tools for A/B Testing Web Products 🧙‍♂️
  • A/B Test Case Studies 🤓
  • Conclusion 🙌
  • Important disclosure: we're proud affiliates of some tools mentioned in this guide. If you click an affiliate link and subsequently make a purchase, we will earn a small commission at no additional cost to you (you pay nothing extra). For more information, read our affiliate disclosure.

    What Is A/B Testing? 🤷‍♀️

    A/B testing (also called split testing or bucket testing) is a type of marketing experiment where two or more versions of something (a product, website, email, etc.) are simultaneously sent to different audience segments to determine which variation performs better.

    For example, you have a product launch email you want to send out but are not sure which button size will work best for your call-to-action (CTA). You create two variants and send the email out to your audience segments to see which button results in the most clicks and conversions. Once sent, you gather the results and can determine which CTA button worked best.

    💡
    While mostly associated with digital marketing and products, A/B testing has been around since long before the internet. The modern version we know today was first used by Google in 2000 to identify how many search results users wanted to see on one page.

    Today, it's a tool used by most companies and its popularity is owed to the fact that businesses can gather user preferences, not through words that can be sugar-coated, but by their actions. This makes it a super reliable method of gathering insight into your user preferences.

    The only 'rule' is to test a single element at a time or the data you gather won't be specific enough to determine which element garnered the most success.

    The two main notions that rule in the A/B testing world are:

    1. Control – the original version of the element you want to test;
    2. Variation – the alternative, such as a different button color, website color scheme, marketing email subject line, time of day you send out a promotional post, etc.

    The control and variation need to be tested simultaneously to accurately compare their effectiveness. If tested separately, there could be factors that impact your user behavior and might change the results and provide inaccurate data.

    Types of A/B Testing 🕵️‍♀️

    Split testing gives you insight into how well (or not) you know your target audience or how subjective your opinions/preferences might be. You might think that a video in your email campaign will outperform a photo but when tested, the photo actually outperforms the video by 35%.

    The above example shows that you can't just trust your instincts when it comes to the web. You need to test your hypothesis to make sure you're on the right track - and you might be surprised by the results!

    There are three types of A/B testing you can use:

    1. A/B split testing: This is a regular experiment where an original element (or a cluster of elements) from a web page is tested against its modified version. If you involve several variations, the element(s) under the change must stay the same.
    2. Multivariate testing: If you want to assess a call-to-action button in the first variation, replace the background photo in the second one, and rewrite the subject line in the third, this type of testing is just for you. You will be able to see which element increases your conversion rate the most in comparison with your control variant(s).
    3. Experimental design: This is almost a combination of the above two methods. It allows you to evaluate several changes in one variation with completely different modifications in another one. For instance, your first variation has an amended CTA button and a different background photo, while your second variation tests a rewritten subject line and more white space between product descriptions.

    Naturally, each of the above has its pros and cons. For example, split testing limits you to one element or the same set of elements that can be tested, but the results are super easy to interpret.

    Multivariate testing, however, gives more variety but requires a larger number of users because you'll want to test each element on a different audience group and will need to wait to gather scalable results.

    Experimental design is great in that reduces the number of variations but it's harder to interpret the data collected.

    It's up to you to choose the type of A/B testing that will suit your unique situation. If you're still unsure about applying these methods, read on to find out why this is one thing you really shouldn't skip out on.

    Why Is A/B Testing Important for Your Web Product? 💸

    Before we dive into the details of the A/B testing process, we want to share the benefits of this testing program with you to help you understand the value behind it.

    • A proven method of accessing honest feedback. Users don't know that they are participating in a testing campaign so they just naturally choose what they prefer.
    • It's universally applicable. Whether you're a small business or a large corporation, or even an individual designer/developer, this method will work. It also works with any size database.
    • It’s low-risk. You won't end up losing tons of money by testing elements with this process. You test and then gradually introduce changes that are already “approved” by your end users.
    • Reduce bounce rates. It's a great way to identify what exactly makes visitors leave your website/page.
    • Boost conversion rates. By improving buttons, calls to action, and overall design/user experience/pieces of content/etc. you are far more likely to see an increase in conversions.
    • Optimize the work of the team. It is no longer necessary to spend hours determining which version of the text or which background color is better. Your potential customers show you exactly what works for them.

    Now let’s find out more about the actual process.

    A/B Testing Guide 📝

    To gain the maximum benefit from split tests, you need to use it for the specific elements it was designed for. We're going to guide you through the testable elements and expand on how they are analyzed using this method.

  • Layouts, Colors, etc.
  • Headlines and Product Descriptions
  • Calls to Action and Buttons
  • Pricing and Offers
  • Images
  • Amount of Text
  • Subscription Forms
  • Layouts, Colors, etc.

    You can be a stellar designer or have an award-winning design team, but at the end of the day, it's the user who makes the choices. This is why, when creating a website, blog, or landing page, etc., it's so important to know which design leads to the highest engagement and conversions from your visitors.

    You can perform various tests on background colors, website elements, menu structures, and text.

    It's a good idea to test not only big components but also small variations because the details play an important role in visitor behavior.

    Headlines and Product Descriptions

    You can use A/B testing to determine which title entices readers to open your article and what kind of product description persuades your potential customers to make that purchase. Content and headlines have far more impact than most people realize!

    To get the most out of this, brainstorm 10 - 15 samples of the same headline or short description, select your top 3 or 4, and put them to the test. The stats will show which type of content will get you closest to your conversion goal.

    Calls to Action and Buttons

    If you need to know which sign-up button gets the most clicks or which call to action receives the most attention, then A/B testing will point you in the right direction. You can find out what shape, color, or location generates the highest conversions.

    It's often the smallest details that can make a big difference. You might find that a "Learn more" call to action attracts far more attention than the usual "Sign up". But you won't know until you test it.

    Pricing and Offers

    We might think that the lower price generates the most sales but that isn't always true. Other factors such as product quality, color options, social proof, customer experience, etc. all play a role in the purchase decision.

    You can use A/B testing to evaluate sales pages and offers to establish where your best sellers are.

    Images

    It's critical to test images for marketing campaigns, homepages, sales landings, and online stores. You might discover that having a full 360-degree picture drives more sales than a carousel or that a colorful image on your home page actually gets more attention than you thought it would.

    When assessing images, remember to test them against other variations as well as your control version.

    Amount of Text

    It's not just about what you're saying, it's about how you say it. Test content depth to determine whether your readers prefer longer reads or smaller, snappier blog posts. You can also find out how much text to put into one line to make it more readable.

    Subscription Forms

    There is a lot that goes into a subscription form; wording, form fields, colors, font, etc. These all make a difference to the subscriber who is deciding whether to sign up or not. For conversion rate optimization, it's best to test exactly which elements create the perfect form.

    The above are some of the main elements but there are additional variants you can consider. Whatever you choose, it's essential to focus on the single element that requires changes then create several variations and analyze the number of subscriptions that come in.

    How to Conduct A/B Testing 👩‍🏫

    Yes, it's as easy as the above image makes it look!

    To obtain the best results from this approach, you have to create a solid strategy that includes the following:

    Defined objectives

    A/B testing is a scientific approach that needs an evidence-driven approach. There are several ways to understand your goals and support them with proof:

    Technical analysis
    Evaluation of web analytics
    Mouse-tracking analysis
    User testing
    Surveys

    By using these techniques, you'll be able to isolate the weak points and clearly formulate a thesis. For example, the CTA button must be changed for the new version to generate more clicks.

    Variations

    You obviously need variations to test what works but keep it simple. If you test too many variations at the same time it may result in distorted findings because your audience will be split into too many groups.

    Redirections

    If you want to test a separate element you need to create an HTML version of the variation and upload it to your A/B testing tool. Alternatively, you can create a variation within the software you use which will generate a code for you to insert into the website.

    The tool will replace the control with the variation before a random user accesses the website.

    If you wish to test the whole page, you need to develop and upload it to your website. This way, your A/B tool will redirect some visitors to a modified variation.

    Time limit

    You will need to determine the duration of your test up-front. Experts recommend that the test should last at least one week. If you plan to run your test for more than a week, take into account that roughly 10% of users delete cookie files every two weeks.

    Also, keep in mind that the more variations you test simultaneously, the longer the experiment should be.

    Once you have those in place it's time to A/B test. Choose the most suitable tool and run as many tests as you have resources for.

    After your set time frame, it's time to analyze the results. Consider metrics such as percentage increase, confidence level, direct and indirect impact on other metrics, and so on. To dig deeper it's often best to export your results to an Excel document or plug your results into a testing calculator. This will structure your data and make it much easier to analyze.

    Researchers advise that you integrate your A/B test tool with Google Analytics to get more refined results.

    Each case is unique, so there might be an additional transitional step, but in general, the process of A/B testing looks like the above.

    Now let's look at the software and tools that you'll need to perform the perfect split test.

    Tools for A/B Testing Web Products 🧙‍♂️

    There is no shortage of tools available when it comes to A/B testing. Let's take a look at some of the options out there.

    A/B Testing Platforms

    There are fantastic software options that can handle the full experiment. The most popular tools are Optimizely, VWO, Convert, AB Tasty, and Unbounce. To use these tools you just need to register your website, create a variation, and paste the code you receive into your website - that's it! Testing platforms will track your users' behavior and show you the results at the end of the experiment.

    Social Networks

    Yes, you read that correctly. Social media is a great tool and Buffer actually successfully tests its blog headlines on Twitter. They simply post different titles for one article and look at which one gets the most attention (retweets, mentions, favorites, and clicks) to choose the most engaging one. This can be replicated on Facebook.

    Inbuilt A/B Testing Tools

    Various platforms that are used for different purposes have their own built-in A/B testing tools. Some of the more well-known ones are MailChimp, WordPress plugins, and Facebook Manager ad campaigns. There are plenty of these around and so look around to find the most suitable option for your needs.

    A/B Test Case Studies 🤓

    Let’s look at some A/B testing examples conducted by well-known companies and consider the results:

    Creative Market (Design Platform)

    The Creative Market team set a business goal of increasing conversions by redesigning their pricing offers page for credit purchases. They created a variation relying on the best practices of usability: reducing an additional click and placing a CTA button above the fold. What do you think happened?

    They found that the best practices actually didn’t work! With the variation design, total revenue dropped by 11%.

    Fab (Platform For Buying And Selling Handcrafted Goods)

    The Fab team used split testing and discovered that adding a CTA text to a cart button increased the click-through rate (CTR) by 49%. That proves the general hypothesis that users react better to text than to icons.

    Omoda (Dutch Shoe Retailer)

    The design team at Omoda believed that their unique selling points were not sufficiently visible in the current version of their design. VWO proposed that they increase their visibility by changing the background of customer service elements to grey and developing unique icons for each USP.

    While the desktop version didn't cause much of a stir, they saw sales from mobile devices blow up by 13.6%.

    These case studies prove that user behavior isn't always what you think it will be! A/B testing is the ultimate method for determining what your customers really like and what drives the best conversion rates.

    Conclusion 🙌

    Accurate A/B testing can really make a huge impact on your return on investment. From decreasing your cart abandonment rate on your online store to boosting your marketing strategies and gaining more long-term customers, improving your user journey, or simply giving your customers a product experience they will never forget, split testing can do it all!

    Using the knowledge gained from our guide, we hope you will now be able to set solid goals, choose the right testing technique, and successfully run your experiment.