Running an A/B test without knowing your required sample size is like driving blindfolded. You might reach your destination, but you'll probably crash. This guide will teach you exactly how many visitors you need for reliable test results.
Why Sample Size Matters
Underpowered tests are the #1 cause of false conclusions in A/B testing. Here's what happens:
Too Few Visitors
- • Results swing wildly day-to-day
- • False positives look like winners
- • Real improvements go undetected
- • You make wrong decisions
Right Sample Size
- • Results stabilize and converge
- • True winners are identified
- • You can trust your data
- • Decisions drive real growth
The Sample Size Formula
Sample size depends on four key inputs:
1. Baseline Conversion Rate
p₁Your current conversion rate. Lower baselines need more samples. Example: 5% signup rate.
2. Minimum Detectable Effect (MDE)
δThe smallest improvement worth detecting. Smaller MDE = more samples needed. Example: 20% relative improvement (5% → 6%).
3. Statistical Significance (α)
α = 0.05Probability of false positive. Standard is 95% confidence (α = 0.05), meaning 5% chance of declaring a winner when there isn't one.
4. Statistical Power (1-β)
1-β = 0.80Probability of detecting a real effect. Standard is 80% power, meaning 20% chance of missing a real winner (false negative).
Simplified formula: At 95% confidence and 80% power, sample size per variant ≈ 16 × p(1-p) / δ² where p is baseline rate and δ is absolute effect size.
Sample Size Reference Table
Visitors needed per variant at 95% confidence, 80% power:
| Baseline | 10% MDE | 20% MDE | 30% MDE | 50% MDE |
|---|---|---|---|---|
| 1% | 156,000 | 39,000 | 17,400 | 6,300 |
| 2% | 77,000 | 19,300 | 8,600 | 3,100 |
| 3% | 51,000 | 12,800 | 5,700 | 2,100 |
| 5% | 30,400 | 7,600 | 3,400 | 1,200 |
| 10% | 14,400 | 3,600 | 1,600 | 600 |
| 20% | 6,400 | 1,600 | 720 | 260 |
* MDE = Minimum Detectable Effect (relative improvement). For a 50/50 split test, multiply by 2 for total visitors needed.
Traffic → Test Duration Calculator
Based on your daily traffic, here's how long tests will take (assuming 5% baseline conversion):
| Daily Visitors | Weekly | 20% MDE | 30% MDE | 50% MDE |
|---|---|---|---|---|
| 100 | 700 | 8 weeks | 4 weeks | 2 weeks |
| 500 | 3,500 | 2 weeks | 1 week | 4 days |
| 1,000 | 7,000 | 1 week | 4 days | 2 days |
| 5,000 | 35,000 | 2 days | 1 day | <1 day |
| 10,000 | 70,000 | 1 day | <1 day | <1 day |
Important: Always run tests for at least 1-2 full weeks regardless of sample size to capture day-of-week effects.
What If You Don't Have Enough Traffic?
Low-traffic sites can still run meaningful tests. Here's how:
A 50% improvement needs 1/4 the sample of a 10% improvement. Go bold.
90% confidence requires ~30% fewer visitors than 95%. Acceptable for most decisions.
Test your homepage, not your about page. Concentrate traffic where you can.
Test clicks instead of purchases. Higher baseline = smaller sample needed.
Sometimes the right answer is to wait. A 6-week test beats a wrong conclusion.
Sample Size Mistakes to Avoid
❌ "We'll just run it until we see significance"
This is called "peeking" and inflates false positives to 30%+. Always set sample size before starting.
❌ "100 conversions per variant is enough"
This rule of thumb is often wrong. Required sample depends on your baseline and MDE, not a magic number.
❌ "We got significance after 2 days, ship it!"
Early significance often regresses to the mean. Wait for your pre-calculated sample size AND at least 1-2 weeks.
❌ "We'll detect a 5% improvement"
Detecting small effects requires massive samples. A 5% MDE at 5% baseline needs ~300,000 visitors per variant. Be realistic.
Step-by-Step: Calculate Your Sample Size
Find your baseline conversion rate
Check your analytics for the page/action you're testing. Use at least 30 days of data.
Choose your MDE (be realistic)
20-30% relative improvement is typical. Smaller effects need exponentially more traffic.
Use the reference table above
Find your baseline and MDE, multiply by 2 for total sample (both variants).
Calculate test duration
Divide sample size by your daily traffic. Ensure it's at least 1-2 weeks.
Commit to the plan
Don't stop early, don't peek and make decisions. Wait for your sample size.
Calculate Before You Test
Sample size calculation isn't optional—it's the foundation of reliable A/B testing. An underpowered test is worse than no test at all because it gives you false confidence in wrong conclusions.
Use the tables in this guide, be realistic about your MDE, and commit to running tests for the full duration. Your future self (and your conversion rate) will thank you.