A/B testing seems simple: show two versions, pick the winner. But behind that simplicity lurks a minefield of statistical traps that can lead you to confidently make the wrong decision. Here are 12 mistakes that ruin tests—and how to avoid them.
The Cost of Testing Mistakes
A flawed A/B test is worse than no test at all. Why? Because it gives you false confidence to make changes that hurt your business. These mistakes can:
Ship Losers
Implement changes that actually hurt conversion
Waste Time
Run tests that can't produce valid results
Miss Winners
Reject changes that would have improved results
Statistical Errors (Mistakes 1-4)
Stopping Tests Too Early
You see 95% significance after 3 days and ship it. Two weeks later, the "winner" is underperforming. What happened?
The problem: Early results are noisy. A test showing +30% after 200 visitors often regresses to 0% after 2,000.
The fix: Calculate sample size BEFORE starting. Don't stop until you reach it AND run for 1-2 full weeks.
Peeking at Results Daily
Checking your test every day and planning to stop when you hit significance? You've just inflated your false positive rate from 5% to 30%+.
The problem: Multiple comparisons. Each peek is another chance to see random noise that looks like signal.
The fix: Set a fixed duration upfront. Only analyze results at the end. Or use sequential testing methods designed for peeking.
Underpowered Tests
Running a test with 500 visitors when you need 5,000 is like trying to hear a whisper at a concert. You'll miss real effects.
The problem: Low statistical power means high false negative rate. You'll reject winners as "inconclusive."
The fix: Use a sample size calculator. If you don't have enough traffic, test bigger changes or accept lower confidence.
Testing Too Many Metrics
You track 20 metrics and one shows significance. Winner? Not necessarily—at 95% confidence, 1 in 20 metrics will show false positives by chance.
The problem: Multiple comparison problem. More metrics = more chances for random noise to look significant.
The fix: Designate ONE primary metric before starting. Use other metrics as directional only, not for decisions.
Test Design Errors (Mistakes 5-8)
Testing Multiple Changes at Once
You change the headline, button color, and image. The variant wins. But which change caused it? You have no idea.
The problem: You can't isolate the effect of individual changes. You might ship a winner that includes a loser.
The fix: Test one variable at a time. Or use multivariate testing if you have enough traffic (you probably don't).
Not Running Full Business Cycles
Your test ran Monday-Thursday and showed a winner. But your weekend traffic behaves completely differently.
The problem: User behavior varies by day, week, and season. Partial cycles give skewed results.
The fix: Always run tests for at least 1-2 full weeks. For B2B, consider full months to capture billing cycles.
Ignoring Segment Effects
Overall, your test is flat. But mobile users loved the variant (+20%) while desktop users hated it (-15%). You'd never know without checking segments.
The problem: Aggregate results hide segment-level effects. You might roll out something that hurts your best customers.
The fix: Always check key segments (device, traffic source, new vs. returning) before rolling out winners.
Testing Low-Impact Changes
You spent 4 weeks testing button color (blue vs. green). Result: inconclusive. You've wasted a month on something that rarely matters.
The problem: Low-impact tests need huge samples to detect effects. You'll run out of time before finding significance.
The fix: Prioritize tests by potential impact. Headlines, value props, and pricing beat button colors every time.
Process Errors (Mistakes 9-12)
No Hypothesis
"Let's see what happens if we make the button bigger." This isn't a hypothesis—it's a guess. And guesses don't produce learnings.
The problem: Without a hypothesis, you can't learn why something worked or didn't. You can't apply insights to future tests.
The fix: Write a hypothesis: "We believe [change] will [outcome] because [reasoning]." Document it before starting.
Not Validating Tracking
Your test shows variant B converting at 15% vs. control at 3%. Amazing! Except... your tracking was broken and counting pageviews as conversions.
The problem: Garbage in, garbage out. Bad tracking = bad decisions.
The fix: Test your tracking before launching. Verify conversions are firing correctly on both variants.
Confusing Correlation with Causation
"Users who saw the variant bought more!" But wait—did the variant cause more purchases, or did high-intent users happen to see the variant?
The problem: Without proper randomization, you can't establish causation. External factors may be driving results.
The fix: Ensure random assignment. Check that control and variant groups have similar characteristics at baseline.
Not Documenting Results
You've run 50 tests this year. What did you learn? If you can't answer that question, you're not building organizational knowledge.
The problem: Without documentation, you'll repeat mistakes, re-test things you've already tested, and lose learnings when people leave.
The fix: Create a test repository. Document hypothesis, results, learnings, and next steps for every test.
Pre-Launch Checklist
Before launching any test, verify:
Test Smarter, Not Just More
The goal isn't to run more tests—it's to run tests that produce reliable insights. By avoiding these 12 mistakes, you'll transform your experimentation program from a guessing game into a growth engine.
Remember: a well-designed test that shows no effect is more valuable than a flawed test that shows a false positive. Trust the process, respect the statistics, and let the data guide you.