Your test has been running for two weeks. The confidence level is stuck at 47%. Nothing is happening. Sound familiar? Here's what's probably wrong — and how to fix it.
Quick Diagnosis
Before diving into specifics, check these first:
The 8 Most Common Problems
Not Enough Traffic
Symptoms
- Test running for weeks with no significance
- Confidence jumping between 30-70%
- Results that keep flipping
Cause
Your sample size is too small to detect the effect you're testing for.
Fix
Either run longer, test a bigger change, or focus on higher-traffic pages. Use a sample size calculator before starting.
Testing Too Small a Change
Symptoms
- Variant looks almost identical to control
- Expecting 2-3% lift
- Button color changes
Cause
Small changes need massive sample sizes to detect. A 5% improvement requires 10x more traffic than a 50% improvement.
Fix
Test bigger, bolder changes. Headlines, value props, layouts — not button colors.
Stopping Too Early
Symptoms
- Stopped when you saw 95% confidence on day 3
- Results flipped after you stopped
- Different result each time you check
Cause
Early results are unreliable. Statistical significance can appear and disappear as data accumulates.
Fix
Calculate required sample size before starting. Commit to that duration. Don't peek.
Wrong Goal Tracking
Symptoms
- Conversions not registering
- Numbers don't match other analytics
- Zero conversions recorded
Cause
Goal tracking isn't set up correctly. The event isn't firing, or it's firing for the wrong action.
Fix
Test your goal tracking before launching. Click the button yourself and verify it registers.
Flicker Contaminating Results
Symptoms
- Users see original then variant
- Content flashes on page load
- Variant users behave like control
Cause
Users are seeing both versions, contaminating the test. They're no longer a clean variant group.
Fix
Use a tool with proper anti-flicker implementation. Or add an anti-flicker snippet.
Testing the Wrong Page
Symptoms
- High-traffic page but low conversions
- Goal is 3 clicks away from test
- Testing top of funnel, measuring bottom
Cause
The page you're testing is too far from the conversion event. Too many variables between test and goal.
Fix
Test pages closer to the conversion. Or use micro-conversions (clicks, scroll depth) as goals.
Variant Not Loading
Symptoms
- Analytics show 0 variant visitors
- Variant never appears when you test
- Only control is being served
Cause
Technical issue preventing the variant from loading. Could be script error, targeting issue, or caching.
Fix
Check browser console for errors. Verify targeting rules. Clear cache and test in incognito.
External Factors Skewing Results
Symptoms
- Big spike in one variant mid-test
- Results don't match historical patterns
- Unusual traffic sources
Cause
Something external affected one variant more than the other. Holiday, press mention, ad campaign.
Fix
Check for external events. Segment by traffic source. Consider restarting the test.
Prevention Checklist
Before launching any test, verify:
When to Stop a Failing Test
Most Tests Aren't Broken — They're Underpowered
The #1 reason tests "fail" is insufficient sample size. Before assuming something is broken, calculate whether you've had enough visitors to detect the effect you're looking for.
When in doubt: run longer, test bigger changes, or focus on higher-traffic pages.