How I Come Up With A/B Test Ideas

Critical to the overall success and growth of any website is the ability to test, learn, and improve on the ways in which you serve your website visitors. A/B testing allows web marketers to compare different versions of a webpage or page element to determine which one is better for users. But where do the testing ideas come from? To build a backlog of experiments worth conducting, I collect ideas for testing in 6 ways:

Quantitative Data Analysis

Quantitative analysis allows us to identify user behavior patterns and pinpoint areas where improvements can have the most significant impact. By taking advantage of tools like web analytics (i.e. GA4) and heatmaps, we can assess the performance of various elements on a website, such as page layout, content, and call-to-action buttons. This data helps in hypothesizing which changes might enhance user experience or increase conversion rates.

Example: Conversion rate from the product detail page (PDP) is lower than typical benchmarks. This quantitative insight might lead to the hypothesis that perhaps the CTA isn’t clear, pricing is an inhibitor, or that the page is lacking information that the user needs to proceed.

Qualitative Data Analysis

While quantitative analysis is great for understanding what is happening, it falls short in terms of being able to tell you why. Using qualitative analysis provides a deeper, more contextual understanding of user experiences, motivations, and pain points. Employing methods such as user interviews, surveys, session replays, and usability testing helps generate insights into these "why" behind user behaviors. Qualitative feedback uncovers nuanced perspectives and emotional responses, helping to identify specific issues that users face and uncovering areas for potential A/B testing that might not be evident through numerical data.

Example: Back in 2021, Apple changed the “subscribe” button to “follow” within the Apple Podcasts app. This was based on qualitative survey insights from users that thought to “subscribe” to a podcast meant that it would cost money.

Internal Ideas

Another great way to source ideas for A/B testing is by simply asking your teammates to provide them. This doesn’t have to be limited to folks on the web team either. I’ve also shared website insights and shortcomings with marketing, product, and customer support teams and asked “why do you think this is happening?” People are more than willing to share their thoughts and ideas on why and suggest their ideas for improvement. The hidden upside with testing ideas sourced internally, is that if you run one that’s a winner, that can elevate morale and instill a greater sense of camaraderie between teams.

Example: Customer service teams indicate that they are fielding a lot of inquiries related to the return policy. This insight could mean that more clearly stating the return policy may give users greater peace of mind when it comes to moving forward with their purchase. An added bonus is that it will likely reduce calls to customer care.

Websites

Just browsing other websites probably seems like an obvious way to source A/B testing ideas, but I find it is often underutilized. While most web marketers are at the very least looking at competitor sites, I think it’s also important to look at industry adjacent sites. These would be sites for brands that aren’t necessarily competing with you for the same customers, but are potentially selling a non-competitive product to the same or similar customer profile. I do this on a regular basis, and simply screenshot interesting or unique ideas which I throw into a swipe file.

Example: If you sell say laptops, a high-priced consumer product where performance, style, and durability are paramount, there may be ideas worth “borrowing” and testing from comparable industries such as cars or even certain home goods.

Follow the Experts

There are a number of CRO and experimentation pros out there sharing the results of tests that they’re conducting. While someone else’s successful test can’t just be applied to another website with the expectation of achieving the same results, I consider the validation from the field sufficient to warrant testing it out for myself.

Example: You can find experts sharing insights on social media (LinkedIn and X most notably) and also on blogs, particularly from third-party vendors like Unbounce (landing pages), SearchPilot (SEO testing), or A/B Tasty (Testing Platform).

Be a Little Contrarian

I often find myself inspired by those who think and do things differently, particularly in business. And a core component of effective marketing is effective differentiation. A fun way that I like to source testing ideas is by observing what others are doing out there and looking for patterns. And if there is a particular approach that feels a bit saturated to me, that is an indication that perhaps doing that thing a little differently, entirely differently, or even not at all may be worth testing. The idea here isn’t to just do the literal opposite, but to examine whether standing out among the crowd resonates with your users better.

Example: Your top competitors are all leading with the same feature. That might be an opportunity to pivot to the benefit of that feature or even an entirely different feature altogether.