What 6 weeks of homepage A/B testing taught us

At Contentsquare, our website is never just a website. When you build a product that helps brands understand how their users behave online, your own site becomes your most visible case study. So when we had a hypothesis—that an AI-powered conversational experience could outperform our standard homepage hero—we didn't debate it. We tested it.

For six weeks, we ran a 50/50 A/B test on our English desktop site. One variant was our standard homepage. The other replaced the hero with a live AI chatbot with pre-written prompts and a higher visual product focus. We measured everything using our own platform.

Here's what we found.

Key insights

• +35% lift in session time

  for the AI chatbot variant, exceeding our +20–30% target from day one

• Even a 2% interaction rate can validate a concept Only 2% of visitors reached the chatbot, but those who did spent nearly 2 minutes in high-intent conversation.

• Knowing who's actually on your page changes everything The most-clicked element on our standard homepage was the Login button (22.8% of clicks). Separating existing customer traffic from net-new prospects gives you a much truer read on conversion performance.

• The real fold is ~1,400px, regardless of how good your design is Both variants lost the majority of visitors at the same scroll depth. If your strongest CTAs live below that line, most visitors will never see them.

• AI search is already sending you traffic — even if you're not tracking it ChatGPT drove organic visits to our test page unprompted. B2B SaaS brands that create AI-readable, conversational content are quietly building a new discovery channel right now.

Why we ran this test

Our standard homepage was doing its job. Traffic was healthy, demo requests were coming in, and the brand narrative was clear. But "doing its job" isn't a high bar for a company that believes digital experience is a competitive advantage.

When we looked honestly at the page, we saw a familiar problem: it was built for product-aware visitors. For everyone else—a prospect arriving from paid social with a live business problem and no prior context—it asked them to self-identify, navigate our platform taxonomy, and figure out which solution applied to them. That's a lot of cognitive work before they even reach a CTA.

Our hypothesis: what if, instead of explaining what we do, we asked visitors what they needed?

The AI Hero variant replaced the static homepage hero with a conversational interface. Visitors could choose from pre-written prompts ("How do I prove ROI to my stakeholders?" or "Why should I switch to Contentsquare?") or type their own question. The product would then respond with relevant context and guide them toward a demo request or sign-up.

What the data showed

Engagement: we exceeded our own target

From day one, the AI Hero outperformed our engagement goals. We'd set a target of a +20–30% lift in session time. The variant delivered +35%.

AI Hero users spent less time on the homepage itself (35s vs. 37.1s for the standard), but significantly more time exploring the rest of the site—7 minutes 49 seconds average vs. 7 minutes 6 seconds. That's not a confused user. That's someone who found something relevant and kept going.

Specific interactive elements within the AI Hero page reached a 68% engagement rate—well above the 2026 B2B industry benchmark midpoint of ~55%.

Conversion: the standard homepage won—but not for the reasons we assumed

For the first four weeks, the standard homepage generated more demo requests. The gap started at -34.7% in week one and closed to around -26% by week four. Directionally improving, but still behind.

Our first instinct was that the chatbot was too distracting. But when we dug into the behavioral data in Contentsquare, a different story emerged.

The most-clicked element on the standard homepage wasn't "Book a demo." It was the Login button, at 22.8% of all clicks.

That means a substantial portion of the standard homepage's "converting" traffic were existing customers who had no intention of filling out a demo form. Strip those out, and the conversion gap between the two variants narrows considerably.

Meanwhile, the AI Hero had a 53% activity rate compared to the standard's 23%. Standard visitors were reading and leaving. AI Hero visitors were interacting and exploring.

The chatbot funnel: the most important structural finding

Only 1.9% of visitors to the AI Hero page ever reached the chatbot.

That number stopped us.

Of the small group who did interact with it, the experience was excellent—they spent nearly two minutes in conversation, asking high-intent questions like how to build a business case for UX investment or how Contentsquare compares to alternatives. These weren't casual browsers. They were active buyers.

But 98% of visitors never got there.

The diagnosis: the hero was visually complex. A dashboard screenshot, multiple interactive elements, and a chatbot interface competed for attention in the same viewport. Users saw it, chose not to engage, and either scrolled down or bounced. We'd designed for the curious minority—and underserved the uncertain majority.

This was the most humbling data point of the entire test. A feature can be brilliant for the people who use it, and completely invisible to everyone else.

What our scroll data revealed

Using Zone-based Heatmaps and Session Replay, we discovered something that applied to both variants equally: both pages lost the majority of visitors at the same scroll depth—around 1,400px, roughly 29% of the full page height.

Everything below that line was seen by fewer than 30% of visitors.

This wasn't a finding about which variant was better. It was a finding about the homepage as a format. The battle for attention is won or lost in the first screen—everything else is there for the minority who choose to stay.

An unexpected discovery: ChatGPT as an organic channel

By week three, we noticed referral traffic to the AI Hero page from ChatGPT. Not from paid campaigns—from users who'd asked an AI assistant about digital experience analytics and been directed to our page.

This is a discovery channel that barely existed 18 months ago. Our test surfaced it without planning for it. It's a signal worth watching carefully.

What we learned about testing itself

Running this experiment taught us as much about how to test as it did about our homepage.

Seasonality is real. Week two saw a significant drop in clicks—not because the variant was failing, but because of Memorial Day and EMEA bank holidays. If we'd drawn conclusions at week two, we'd have made the wrong call. Short test windows are dangerous.

Statistical significance takes time. In week one, the demo request difference between variants was 13 vs. 22 submissions. That feels meaningful. It isn't. The volume was too small to conclude anything. Waiting for significance rather than acting on early directional signals is genuinely difficult—and genuinely necessary.

The test became a qualitative intelligence engine. The chatbot logged 582 conversations in 17 days. Visitors told us, in their own words, what problems they needed to solve. No survey. No focus group. Just a live window into real buyer intent—signal that has value far beyond the A/B test itself.

Security is a non-negotiable for every homepage. We logged 24 structured prompt injection attempts in the first two weeks. If your website is becoming an AI surface, your security posture needs to evolve with it.

The answer isn't A or B. It's what A and B taught you.

After six weeks, our conclusion isn't that the AI Hero won or that the standard homepage won. It's that we now know enough to build something better than either.

The standard homepage has strong conversion architecture. Its hero drives clear CTA intent and channels high-intent visitors efficiently. Its weakness: it's essentially a single decisive moment. Visitors who don't click at the top largely disengage.

The AI Hero has stronger engagement quality. It builds intent progressively, educates more effectively, and its mid-page product storytelling is measurably better. Its weakness: the hero creates too much visual complexity with too little immediate clarity about what Contentsquare does—and for whom.

The optimal homepage is a hybrid: the visual urgency and CTA prominence of the standard, combined with the progressive education and interactive depth of the AI variant.

That's what version two will be.

Why this matters beyond our homepage

The most dangerous thing a marketing team can do is optimize for the visitor they imagine, rather than the visitor who actually shows up.

Our test revealed that a meaningful share of our homepage traffic isn't prospects at all—it's existing customers logging in, internal employees, and casual browsers who'll never convert regardless of how good the headline is. Understanding that changes how you interpret every metric on the page.

It also revealed that the visitors we most want—active buyers with a specific problem and real intent to switch—are underserved by a page designed to explain rather than engage. They arrive with a question. They deserve an answer.

Testing is the only way to close the gap between the experience you think you're delivering and the one your visitors are actually having.

At Contentsquare, we build tools to help brands understand that gap. Running this test on our own homepage was a reminder that no one's exempt from it—including us.