9 Product Experimentation Tools for Better A/B Testing

Product experimentation is more than just A/B testing a button's color: it’s a virtuous cycle that starts with measuring what’s happening in your product, finding things to improve and build on, testing out improvements, and learning what worked.

In this chapter of our product experimentation guide, we take you through 9 different experimentation tools to add to your product stack to diagnose, ideate, test, and evaluate optimizations.

The limitations of A/B testing tools (and how to mitigate them) 

Product teams love A/B testing tools—and for good reason—but they’re not without their limitations. On their own, these tools show you which of 2 variations is better at encouraging users to take a particular action, but not why.

To design better experiments and impactful feature updates, you need to understand the deeper motivations behind user behavior and avoid ineffective solutions like guesswork, chance, or defaulting to HiPPOs—the highest paid person’s opinions. 

A/B testing can also be frustrating. There will be times when you invest weeks experimenting on a well-theorized hypothesis, only for it to be disproved. Product teams must look for the learnings in experiments that don’t go the way they’d hoped—but if you’re only collecting quantitative data, this can be easier said than done. 

So what’s the ideal solution? Instead of focusing on A/B test data in a vacuum, you can combine test results with qualitative insights (the kind you get from surveys and heatmaps via experience intelligence platforms like Contentsquare); this ‘best of both worlds’ approach allows you and your team to optimize your product for user experience and business goals.

5 popular A/B and multivariate testing tools 

Here are 5 popular A/B testing tools that let you create, measure, and report on experiments. 

1. Optimizely

Optimizely is a digital experience platform with multiple tools, including web, server-side, email, and B2B commerce experimentation. With Optimizely’s popular Web Experimentation tool, you can run A/B and multivariate tests and personalization campaigns to improve conversion rates and the user experience. 

Pro tip: Contensquare integrates with Optimizely. That means if you’re running an A/B test in Optimizely, you can use Contentsquare to dig into why winning variations are such a hit with users (or why losing variations are missing the mark). You can: 

• Record session replays—video-style recordings of how users behave on each page variation 

• Launch surveys on the pages you’re A/B testing, asking users’ opinions directly

• Collect heatmaps that show where users click, scroll and hover on your test pages 

• Use the Journey Analysis tool—a sunburst-shaped visualization of your customer journey data—to view how the changes you’re A/B testing affect your key user flows

These tools can add invaluable context to your experiment results, allowing you to iterate on unsuccessful experiments, and identify ways to build on your successful ones. 

Contentsquare’s Optimizely integration is bi-directional: data flows both to– and from– each platform   

2. Omniconvert

Omniconvert is an experimentation tool used by product and ecommerce teams to create A/B tests, personalization experiments, pop-ups and overlays, and trigger on-site surveys.

Omniconvert’s main A/B testing tool is called ‘Explore’ and offers advanced segmentation to create experiments for specific user cohorts. You can segment by over 40 parameters, including geolocation, on-site behavior, and traffic source, to run experiments that only target a specific audience. 

💡 Connect Omniconvert to Contentsquare and benefit from the experience intelligence described above.

Collect insightful qualitative data to explain your A/B test results with Contentsquare’s Omniconvert integration  

3. Dynamic Yield

Dynamic Yield is a tool for personalizing your users’ experiences of your site with a very granular list of characteristics, triggers and behaviors. These advanced personalization tools make Dynamic Yield the A/B testing tool of choice for many enterprise customers. 

If your A/B test disproves your hypothesis, Dynamic Yield might help you find a silver lining using its Predictive targeting feature. This alerts you if a particular audience segment seemed to respond better to one of your losing variations than the overall winner. In such a case, you could serve the winning variation to most of your traffic, and the losing one to the group that seemed to prefer it. 

4. VWO

VWO is a conversion optimization platform for creating A/B, multivariate, and split tests. VWO is primarily intended for marketing and conversion rate optimization teams to create test variations without coding using a visual editor. 

They also offer an enterprise-grade tool, VWO FullStack, for product teams at large companies to run server-side A/B tests without impacting performance. VWO FullStack also has a feature rollout tool, allowing you to release new features to small groups and measure user impact before launching it to all customers. 

💡 Here’s how to connect VWO to Contentsquare. 

5. AB Tasty 

AB Tasty is another leading platform for running A/B, split and multivariate tests across all digital surfaces. It offers a what-you-see-is-what-you-get editor for trialing product changes without involving developers. 

Like VWO, AB Tasty offers a feature rollout tool, which helps you de-risk launches by showing them to small groups before they go live to your entire customer base. This helps you catch any important bugs early. 

AB Tasty also offers an AI tool to segment users by the emotions they’re currently experiencing on your site, so you could, for example, run an AB test only on satisfied users. 

4 tools to get test ideas and see why your A/B tests are winning

A/B testing starts with solid hypotheses founded on good insight. These 4 Contentsquare tools help you collect foundational insights so you can understand why winning tests succeed, double down on what’s working, and create a virtuous cycle of data-backed optimization.

1. Heatmaps

Heatmaps give teams an at-a-glance overview of what users are doing in a product, making them a great starting point for developing testing hypotheses. 

Once you’ve run an experiment, heatmaps give you a deeper insight into why one variation outperformed another. Monitor heatmaps for both test variations and compare results to get a clear, visual overview of how click and scroll activity differed. They’re also great to show team members and stakeholders—it’s hard to argue with a heatmap!

Contentsquare heatmaps showing an A/B test

Heatmaps in action: when the team at the fashion retailer, New Look, decided to A/B test user-generated content (UGC) on their product pages, they used Contentsquare Heatmaps to dig into their results. Heatmaps displayed the improved click rate of pages containing UGC in a visual way the whole team could understand. 

2. Session Replay 

Session replays are video-style renderings of the real actions an individual visitor takes as they use your product or website, from entry to exit. Since they measure all visitor actions, replays are great for finding where people hesitate or get stuck, giving you ideas for fixes and improvements to test. 

Replays also help you understand why some experiments don’t result in a winner, and give you qualitative insight from low-traffic tests that may not have a statistically significant sample size. 

Contentsquare lets you filter session replays from any A/B test variant and see what users did throughout their session, and how their collective behavior affected your test results.

You can view session replays for every variant of your AB test 

Replays in action: the team at home appliances company Electrolux used Contentsquare’s Session Replay tool to watch recordings when they noticed a recurring issue: users were struggling to find the products they were looking for with the website’s search bar. 

From observing user sessions, Electrolux learned not just that their site’s search functionality was unintuitive—but also what a more intuitive solution might look like, based on how people tried to use the feature. This insight inspired a successful A/B test. 

As a result of changes made, purchase path conversion rates increased by 28.6% while lead conversion rates grew by 70% from Q1 to Q4 of the same year.

3. Surveys

Surveys are a quick and easy way for teams to collect feedback directly from users. Knowing what real users actually think about and need from your product helps you build experiments around user experience and reveals issues and opportunities that might come as a surprise to your team. 

If you’re using Contentsquare Surveys, you can set up events targeting to trigger surveys within A/B test variations and collect valuable qualitative data during experiments. Using A/B test surveys alongside quantitative data gives you a fuller picture of not just what users did (like where they clicked), but what they thought while doing it (do they understand what a new feature does? Do they like the new color scheme? Is there something they’re missing?).

Surveys in action: package holiday provider easyJet holidays often decides which A/B tests to launch based on Contentsquare insights. Recently, they noticed mobile users weren’t using a particular feature—a tool to add your favorite holidays to a list. 

They were unsure whether this was due to a UX problem or if users simply weren’t interested in this functionality. So, easyJet launched a mobile survey and learned that users did like the feature. Therefore, they decided to invest resources into testing and developing it. 

4. Journey Analysis 

Customer journey maps help you understand the steps your users take, revealing which pages make or break a user’s decision to convert or act on their goals—and therefore, which pages are ripe for A/B testing. 

Contentsquare’s Journey Analysis lets you turn your customer journey data into a visualization the whole team can understand. If there’s a page where your users typically drop out of processes, Journey Analysis will show it—so you can run A/B tests on it.  

For example, perhaps you know from testing that adding shipping information to your product page increases conversions—but you didn’t know exactly why until you spotted, from Journey Analysis, that users were no longer exiting your conversion flows to visit the help center and look for shipping information! 

Contentsquare’s Journey Analysis tool in action 

Journey Analysis in action: retail bank Natwest used Journey Analysis to examine user behavior on the sign-up journey for their youth savings account. They spotted that there was a high drop-off rate on this account’s product page. Clearly, that was the page to work on. 

The team A/B tested a new design that removed the hero image and added more key benefits and information instead. It resonated more with users, and drop-offs decreased. 

FAQs about product experimentation tools