Your first A/B test in 6 steps: 1) Pick a high-traffic page with a clear conversion goal. 2) Form a hypothesis ("Changing X will increase Y because Z"). 3) Create ONE variation. 4) Calculate how long to run the test. 5) Launch and wait for statistical significance. 6) Implement the winner. Start with headlines or CTA buttons—they're high-impact and easy to test.
Who this is for
- Complete beginners to A/B testing
- Marketers running their first experiment
- Anyone who wants a simple, actionable guide
Before You Start
Make sure you have:
- Enough traffic: At least 1,000 visitors/month to your test page
- A clear goal: Know what "success" looks like (signups, purchases, clicks)
- An A/B testing tool: ExperimentHQ is free to start
Step-by-Step Guide
Choose What to Test
For your first test, pick something:
- High-traffic: Your homepage or main landing page
- High-impact: Something that directly affects conversions
- Easy to change: Text or colors, not complex features
Good First Test Ideas:
Headlines
First thing visitors see, high impact
Example: "Sign Up Free" vs "Start Your Free Trial"
CTA Buttons
Direct impact on conversions
Example: Green button vs Blue button
Hero Images
Sets tone and expectations
Example: Product screenshot vs Happy customer
Form Length
Friction vs information trade-off
Example: 3 fields vs 5 fields
Social Proof
Builds trust
Example: With testimonials vs Without
Form a Hypothesis
A hypothesis is a prediction about what will happen. Use this format:
"If I change [X], then [Y] will happen, because [Z]."
Examples:
"If I change the CTA from 'Sign Up' to 'Start Free Trial', then signups will increase, because it emphasizes the free aspect."
"If I add customer testimonials above the fold, then conversions will increase, because social proof builds trust."
Create Your Variation
Critical rule: Change only ONE thing. If you change multiple elements, you won't know what caused the difference.
✓ Good
Change headline from "Welcome" to "Start Your Free Trial"
✗ Bad
Change headline, button color, AND add testimonials
In ExperimentHQ, use the visual editor to click on the element and make your change. No coding needed.
Calculate Sample Size
You need enough data to trust your results. Use this rough guide:
| Current Conversion Rate | Visitors Needed (per variant) |
|---|---|
| 1% | ~30,000 |
| 5% | ~6,000 |
| 10% | ~3,000 |
These numbers assume you want to detect a 20% relative improvement. Use our sample size calculator for precise numbers.
Launch and Wait
Start your test and resist the urge to peek. Looking at results early and stopping when you see a "winner" leads to false positives.
⚠️ Important: Run your test for at least 1-2 full weeks, even if you reach sample size earlier. This accounts for day-of-week variations.
ExperimentHQ will tell you when your test reaches statistical significance (95% confidence).
Analyze Results
When your test is complete, you'll see one of three outcomes:
✓ Variant Wins
Implement the change permanently. Document what you learned.
✗ Control Wins
Keep the original. Your hypothesis was wrong—that's valuable learning!
○ No Significant Difference
The change doesn't matter much. Test something with bigger impact.
Beginner Mistakes to Avoid
❌ Stopping too early
Don't stop when you see a "winner" after 2 days. Wait for statistical significance AND at least 1-2 weeks.
❌ Testing too many things
Change one thing at a time. If you change headline AND button AND image, you won't know what worked.
❌ Testing random ideas
"Let's try a purple button" isn't a hypothesis. Have a reason for your test based on user research or data.
❌ Not documenting results
Write down what you tested, what happened, and what you learned. Build institutional knowledge.
Ready to Run Your First Test?
ExperimentHQ makes it easy. Visual editor, automatic statistics, and a free tier to get started.