A website interface or an app plays an important role in any marketing strategy. A good-looking landing page that engages and communicates the right message can sell almost anything. But what if it doesn’t? To answer this question, businesses have been testing and verifying their design assumptions for years.
There is a great number of UX testing types, but the most popular way to test your design is to apply A/B and multivariate testing. These approaches are used to compare different variations of the same design or content to evaluate which one leads to better conversion rates. Usually, the testing covers headings, subheadings, media-content (images, video), copywriting, call-to-action buttons (CTAs), fonts, colors, and links. However, you can run A/B tests for nearly every feature.
In this article, you will learn more about the difference between methods of multivariate testing and find out about the best tools to perform this analysis on web and mobile interfaces.
A/B testing vs. multivariate testing vs. split testing
There are three main types of UX variant testing: A/B, multivariate, and split URL. All these entail showing different variations of the same page or product to subsets of users to understand which one works best.
In A/B testing only two versions of an interface are shown. Usually, you compare version A and version B that differ in a single tested item, like the placement of a banner or CTA, font, or heading. Sometimes versions A and B are called the champion and the challenger. The champion is the current variation, while the challenger is the version assumed to perform better. The one that leads to better metrics is chosen as the winner and gets to be leveraged on the final version of a product or website.
A/B testing
Multivariate testing. While A/B testing assesses a single change, multivariate testing evaluates changes of multiple elements and their combinations. For example, there are three types of headlines and two widgets of different colors. A tester creates six variables, and the traffic is split between them. Then the tester can define which of the six combinations performed better. But this type of testing is more relevant to websites with a large volume of traffic.
Multivariate testing
Split URL testing. Sometimes A/B testing and split URL testing are thought to be the same approach. They are not. A split URL test is more complicated and includes server-side changes, meaning that there are two different web pages that are tested against each other. This type usually works for landing pages, when the design team decides which one will lead to better results.
Comparing A/B Testing Tools
Today, there are numerous evaluation tools to run all three types of assessments. Usually, this software is capable of at least A/B testing. Most of the tools offer wider feature-sets and allow for multivariate, split URL testing, and even some simultaneous quality assurance approaches. In this article, we are going to compare A/B testing tools and learn more about their functions.
A/B testing tools compared
Even though all tools have their unique and overlapping features, they can be divided into two broad categories: integrated products and landing-page builders.
Integrated products. These must be connected to your existing landing page or any interface where you can make some tweaks and run experiments.
Landing-page builders. The builders, besides A/B and multivariate testing, reporting, and personalization capacities, have interfaces to… well, build your landing page from scratch, usually with no coding expertise needed.
Please note that this division isn’t precise as one can invent many ways to group them (e.g. by personalization support). However, we consider the availability of a builder a critical differentiation point for most people who consider running their experiments. So, if you consider website redesign and plan on running some experiments, have a look at our top 10 tools.
Integrated products
Google Optimize: A free platform with Google services integration
Google Optimize is one of the most popular A/B testing tools. It’s a part of the Google Marketing platform and it’s free. Optimize allows marketers and designers to run multivariate and split URL tests of websites and apps. It has native integrations with Google services, including Analytics, AdWords, BigQuery, Firebase, and AMP (Accelerated Mobile Pages).
Optimize also has a paid version, Optimize 360. In Optimize 360, a user can select up to 10 experiment objectives – up to three in the base version – and set up about 100 simultaneous experiments, while the base version allows you to run no more than five. You have to get Optimize 360 pricing directly from Google representatives.
Google Optimize's tag manager allows for inserting and easily managing code snippets, tags that are used to collect different information, sending it over to other tracking software on your website (e.g. Hotjar). Optimize comes with an extension for Google Chrome and a WYSIWYG (“What you see is what you get”) visual editor. The editor allows a user to see the final version of a layout while working with it. Generally, Google Optimize would be a great tool for those who already use Google ecosystems to track their web results.
Google Optimize interface Source: Google Marketing Platform
Pros of Google Optimize:
Integration with Google Analytics. This tool is a Google product, and it means that it easily integrates with other Google-powered tools. As Google Analytics is extremely popular, this seamless integration is warmly welcomed by the community.
Interface. Another benefit is a familiar interface. If you’ve ever tried Google Analytics or AdWords, you know what to expect. It’s simple to use and intuitive.
Toolset. Even the free version has all the necessary tools to conduct insightful experiments. The enterprise package doesn’t change the feature set. It provides a larger number of experiments, better collaboration, and individual customer service.
Cons of Google Optimize:
Visual editor. Currently, Optimize doesn’t support drag-and-drop manipulations. On top of that, you can’t upload your image directly. Instead, you have to host your own image and edit HTML where the image must be placed.
Lack of customer support in the free version. Although customer support is available in the free version, users complain that it’s hard to get any answers.
Optimizely: Full-stack experimentation platform for media and large enterprises
Optimizely is a testing tool for websites and apps. It offers solutions for eCommerce, travel, media, finance, and tech companies. With Optimizely's Open source tvOS and Android TV SDKs, users can adapt testing for a big screen on Android or iOS TV sets. The tool supports full-stack experimentation that includes front end and backend changes in testing. A multivariate testing feature allows for multipage testing as well. Optimizely’s visual editor suggests a drag-and-drop flow, which greatly simplifies the changes to the tested pages.
Additionally, the tool has an advanced targeting feature to segment the audience using multiple parameters such as time, geography, or on-site behavior. As Optimizely supports testing on big screens, this tool would be a good solution for app and media product experiments.
Pros of Optimizely:
A wide variety of trackable parameters. Optimizely tracks a whole lot of various user behavior characteristics including video views, button clicks, time on page, and more.
Multiple testing management. Not only does Optimizely provide multivariate testing, it’s also easy to set multiple tests with different audiences simultaneously.
Cons of Optimizely:
Page load time. Many customers report that they experienced problems with page previews as complex and heavy web pages take way too much time to upload. Also, a user can experience problems with element placement after a webpage is uploaded. (To learn more about website speed optimization, consider reading our dedicated article.)
Lack of reporting tools. Optimizely's analytical engine lacks a number of important reporting features, such as lead values, and some others.
Price. Some Optimizely customers find the price of the subscription too high. Interestingly, you have to specify prices directly and they may be adjusted depending on the type of client you are.
VWO: Optimization and funnel analysis solution
VWO is an A/B testing tool that supports multivariate and split URL testing. Editing functions are represented with the WYSIWYG interface, advanced HTML and JavaScript editing, and CSS changes. This tool has a Conversion Optimization Platform with funnel evaluation and visitor behavior analysis. The latter includes click heatmaps and other click reporting.
Clickmap interface Source: VWO
VWO`s reporting interface comes with a revenue report and segment analysis. VWO is mobile-friendly and would be a great choice for A/B testing of iOS and Android mobile applications. Besides that, VWO offers powerful targeting support, including behavior and geo-targeting.
Pros of VWO:
Easy test management. Tests can be easily created and modified with a visual editor. Even beginners can start working with it without additional training. And you can use code editing if needed.
Customer behavior analysis. The tool supports heatmaps and offers over 15 types of targeting options, which ensures highly detailed segmentation.
Cons of VWO:
Conversion goals. Setting multiple conversion goals takes much time and effort, and it's impossible to track whether the users came from a mobile or a desktop version.
Geo-targeting is not available in cheaper plans. Geo-targeting is unlocked with the Enterprise plan only, while most of the customers consider it to be critical for the base plan.
Qubit: CRM integration for travel and retail
Qubit is a testing tool for retail, travel, and eGaming platforms. Among their clients are NET-A-PORTER, Emirates and Ladbrokes Coral. Qubit allows users to set conversion goals and analyze revenue. It also employs AI to verify A/B and multivariate testing results to identify false positives among user actions that lead to conversion. “Our main use cases include enabling companies to connect multiple data sources, segment their website visitors and surface experiences based on the intent and behavior of those customers,” notes George Barker, Communications & Customer Advocacy at Qubit.
The tool supports personalization allowing marketers and designers to set segmentation and target different interface elements and information to different audiences. Qubit collects data using its own protocol or can be integrated with a CRM via API. Other features include visitor analytics, business intelligence, and tag management.
The company also offers a mobile solution, Qubit Aura. “With platforms like Instagram and Facebook building curated experiences around the individual, powered by AI, we've developed the solution for e-commerce. Qubit Aura builds an experience for the user, on their mobile phone, that within a couple of taps, swipes and selections surfaces the most relevant products in real-time,” adds George.
Qubit Dashboard Source: Qubit
Pros of Qubit:
Precise segmentation. The use of machine learning algorithms improves segmentation making it more sophisticated, which leads to better personalization.
Tag manager. Qubit's tag manager works like Google Optimize’s and greatly simplifies the work with third-party integrations.
Cons of Qubit:
Steep learning curve. Qubit is not the tool for beginners, and the vendor does not provide user guides or tutorials. So, it will take some time to learn it.
Customization. You can’t set custom traffic analysis in an A/B test. Also, the number of testing goals is limited compared to other testing tools.
Maxymiser: Mobile-ready cloud solution with cross-channel personalization
Maxymiser is Oracle's cloud-based testing and user experience personalization solution. It provides tools for A/B and multivariate experiments and uses predictive analytics to enable personalization on webpages.
Maxymiser automatically creates data-based visitor profiles to allow for tailoring specific content to particular users. The core features of Maxymiser include personalization, reporting and statistics, cross-device campaigns, and a visual editor that works for desktop web, mobile web, and apps. Maxymiser is mostly oriented to online stores, so it would be a good choice for retail providers.
Mobile campaign interface Source: Maxymiser
Pros of Maxymiser:
Customer support. The support team is known to be really customer-oriented and quick to react to developing technical issues.
Analysis and reporting. The analytic tools of Maxymiser provide deep insight into testing results displayed in reports. A reporting dashboard has a precise filtering engine.
Cons of Maxymiser:
Learning curve. Customers complain about the lack of training for developers. Although the company provides onsite training, these aren't available in all countries.
Data export. Some users find report export a bit challenging and time-consuming.
Freshmarketer: Conversion rate optimization suite with direct visitor feedback
Freshmarketer is a tool for A/B and split URL testing, positioned as a “Conversion Rate Optimizer Suit.” In terms of analysis, the product has heatmaps, session replays, and funnel tracking. You can also personalize user experience by segmenting the traffic and targeting different content to different subsets of it.
Freshmarketer is an approachable, intuitive-to-use interface that doesn’t require special training. The tool integrates with multiple services, offering the Google Chrome extension to facilitate real-time updates.
Dynamic heatmap Source: Freshmarketer
Pros of Freshmarketer:
CRO. Form analysis, heatmaps, and session replay recordings work well and allow for creating custom sales funnels.
Split testing. Freshmarketer has custom attributes for URL split testing; however, some customers find it rather complicated for beginners.
Cons of Freshmarketer:
Support. Customers complain that support is slow.
Technical problems. It can be hard to create a test for the first time and users report experiencing various technical and performance problems.
Landing page builders
Unbounce: A landing page builder with dynamic text replacement
Unbounce is a niche A/B testing tool for landing pages. Apart from being an experimentation-oriented product, it's a landing page builder for marketing campaigns aimed at improving lead generation. The landing page builder feature allows designers to create pages using existing, ready-made customizable templates without coding. The drag-and-drop interface enables adding and moving forms, popups, sticky bars, and lightboxes on a page.
Landing page builder Source: Unbounce
An Unbounce conversion module has a dynamic text replacement feature that allows customers to change keywords across the landing pages for PPC (pay-per-click) campaigns. The tool offers integration with Google AdWords, WordPress, Google Analytics, MailChimp, etc. When working with mail campaigns, the tool provides flexible newsletter tweaking options to customize your eCommerce mailers.
Pros of Unbounce:
Landing page builder. The functionality of the landing page builder allows customers to create a landing page from scratch or build one using a template.
Responsive design ready. Landing pages can be adapted to mobile devices with the Layout Assistant feature that allows users to adjust a desktop landing page for mobile devices.
24/7 customer support. Users can get customer support in five languages via live chat, email, or phone.
Cons of Unbounce:
Learning curve. Unbounce has a complex visual editing system that will take some time to learn, as the interface lacks intuitivity. Also, customers complain that there’s no tutorial.
Instability of the mobile version. Sometimes, video and images are not uploaded or don't work properly in the mobile version.
Instapage: Over 200 website templates for digital marketing campaigns
Instapage is an A/B testing tool and landing page builder. It doesn't support multivariate testing. Similar to Maxymiser, Instapage is fully cloud-based. Key features of Instapage include social network widgets, dynamic custom-code editing, Google Fonts and Adobe Typekit integration, heatmaps visualizer, and a visual editor with a drag-and-drop interface.
Instapage has a vast number of templates for landing pages and its own library of stock images with 33 million photos. Images and videos are easy to manage and download. The tool integrates with third-party providers, including the main email services (MailChimp, ConvertKit, ActiveCampaign).
Landing page builder interface Source: Instapage
Pros of Instapage:
Codeless capabilities. Instapage offers almost 200 different professional landing page templates with plenty of media and fonts to choose from.
Interface. Instapage provides an intuitive flow to create pages, add sections, and move pictures and widgets.
Cons of Instapage:
Editing problems. There are multiple, annoying editing constraints. For instance, you can’t edit multiple visual items at once, e.g. change the color of multiple buttons simultaneously. And, it’s impossible to apply different fonts for a heading and a paragraph.
Support. Users report that customer support doesn't work fast enough.
Limited integrations. Instapage offers third-party integrations, but the number of them is small.
Pagewiz: Real-time CRM integration and lead management
Pagewiz is an A/B testing tool, landing page builder, and lead manager for eCommerce websites. Pagewiz integrates with third-party services including Google Analytics, Google Webmaster, multiple email services, and SEO plugins. Leads are exported to a user's CRM base and updated in real-time. Users can build landing pages from scratch or use existing templates. You also can add external widgets like Google Maps on websites.
Pagewiz allows for extending the functionality of existing pages with HTML and JavaScript code. Main features of the tool include a WYSIWYG drag-and-drop interface, mobile editor, survey feedback, Pagewiz API, statistics, multiple client account management, protocol-based lead export, and lead management.
Pagewiz landing page template Source: Pagewiz
Pros of Pagewiz:
CRM integration. Landing pages that are built in Pagewiz integrate with users' CRM bases, and leads are conveniently exported.
Support and training. The tool offers guides for users in written form (knowledge base and blog), and video tutorials. Customer support is available via multiple channels that include live chat, email, and support center.
Cons of Pagewiz:
Customization. Unlike most of landing page builders, Pagewiz has very limited customization options so you mostly have to rely on existing templates.
Price. Enterprise plan costs start at $999/month.
AB Tasty: Session recordings, contextual targeting, and NPS (Net Promoter Score) tool
AB Tasty is a testing tool and a CRO (conversion rate optimization) platform with personalization support. AB Tasty works with websites, mobile apps, other analytical tools, and integrates with content management systems (CMS). To run a marketing campaign, you can choose one of the existing templates or configure your own. It supports A/B and multivariate testing, heatmaps, and user session replays in a CRO module, and suggests visual and advanced HTML, CSS, and a JavaScript editor. The tool allows for running funnel testing across multiple pages.
Targeting in AB Tasty has two options: standard (by URL or a traffic source), and advanced targeting (based on events, geolocation, etc.) You can also get exportable and custom reports. To verify the data, AB Tasty suggests a statistical reliability indicator.
Pros of AB Tasty:
Content variety. AB Tasty contains a great number of visual features and plugins that allow users to create banners and pop-ups and personalize user experience.
Funnel analysis. The Page Flow feature overviews visitor actions after seeing a particular variation.
Cons of AB Tasty:
Limited heatmapping. Heatmaps cover only specific sections of a page, making it hard to grasp the full picture.
Learning curve. Users without coding experience may find it difficult to use the tool as it sometimes requires code editing. Setting campaign targets may also be difficult for newcomers.
Reporting. The reporting system doesn't analyze all segment combinations.
Final Word
Choosing suitable software may be a tricky task, considering the variety of testing tools on the market, price, and features. The chosen tool should suit the type of business and its main requirements. For example, the features that satisfy a small startup won't be enough for an enterprise. To sum up, here are some other tips for choosing an A/B testing tool:
1. Consider the skills of your current team, because some tools require more advanced knowledge of coding or design. You may consider engaging a UX/UI consulting team to streamline the process. 2. Determine whether you need multivariate or split URL testing for your website. 3. Consider additional features that this tool has. What do you need for your business: a landing page builder, a lead manager, or both? 4. Pay attention to the testing tool`s ability to integrate with other tools, especially analytical ones.