A/B Testing Guide: How to Run Your First Experiment

A/B testing is how the best growth teams make decisions. Instead of debating whether a blue or green button converts better, they test it. Instead of guessing what headline works, they measure it. This guide walks you through everything you need to run your first experiment — from hypothesis to analysis.

What Is A/B Testing?

A/B testing (also called split testing) is showing two different versions of a webpage to different visitors and measuring which version performs better. "Version A" is your current page (the control), and "Version B" is the modified version (the variant).

Traffic is split randomly — typically 50/50 — so each version gets a statistically comparable audience. After collecting enough data, you analyze which version had a higher conversion rate (or whichever metric you're tracking).

Why A/B Testing Matters

Without testing, product decisions come down to opinions — the loudest person in the room wins. A/B testing replaces opinions with evidence:

Step 1: Form a Hypothesis

Every good experiment starts with a hypothesis — a specific, testable prediction about what will happen and why. A strong hypothesis follows this format:

Here are examples of strong vs. weak hypotheses:

The "because" is the most important part. It forces you to think about why a change would work, which helps you learn even when tests lose.

Step 2: Choose Your Primary Metric

Before running any test, decide on one primary metric that determines success. This is your North Star for the experiment.

Step 3: Calculate Sample Size

Before launching, estimate how many visitors you'll need. This prevents two common mistakes: stopping too early (false positives) or running too long (wasting time).

The sample size depends on three factors:

Quick rule of thumb: For a typical SaaS signup page with a 3% conversion rate, detecting a 10% relative improvement (3.0% → 3.3%) requires roughly 25,000 visitors per variant — or 50,000 total.

Step 4: Set Up Your Experiment

There are two approaches to implementation: no-code (visual editor) and code-based. For your first test, we recommend the no-code approach.

Option A: No-Code with a Visual Editor

Tools like ExperimentHQ, VWO, and Optimizely offer visual editors that let you modify your page without writing code. Here's the setup process with ExperimentHQ:

<!-- ExperimentHQ A/B Testing Snippet -->
<script async
  src="https://cdn.experimenthq.io/snippet.min.js"
  data-site="YOUR_SITE_KEY">
</script>

Option B: Code-Based A/B Testing

For more complex changes or single-page apps, you can implement tests in code. Here's a minimal JavaScript example:

// Simple client-side A/B test
const variant = Math.random() < 0.5 ? 'control' : 'variant';

if (variant === 'variant') {
  document.querySelector('.hero-headline').textContent =
    'Start Growing Your Revenue Today';
  document.querySelector('.cta-button').textContent =
    'Get Started Free';
}

// Track which variant the user saw
analytics.track('experiment_viewed', {
  experiment: 'homepage_hero_test',
  variant: variant
});

Step 5: Run the Experiment

Once launched, the hardest part is being patient. Here are the rules for running a clean experiment:

Step 6: Analyze Results

When your test reaches the target sample size, it's time to analyze. Here's what to look for:

Statistical Significance

Statistical significance tells you the probability that your result is real — not just random chance. The industry standard is 95% confidence (p-value < 0.05). This means there's only a 5% chance the difference you're seeing is due to random variation.

What to Do with Results

5 Mistakes That Invalidate A/B Tests

What Should You Test First?

Focus your first tests on pages with the highest leverage — where a small improvement has the biggest revenue impact.