A/B testing comes with its own vocabulary. Statistical significance, p-values, conversion lift — it can feel like learning a new language. This glossary explains every term in plain English.
# Basics
A/B Test
An experiment comparing two versions (A and B) to see which performs better. Half your visitors see version A, half see version B.
A/A Test
A test where both groups see the same version. Used to validate your testing setup and ensure there's no inherent bias.
Baseline
The original version (control) against which variants are compared. Your current conversion rate before changes.
Control
The original, unchanged version in an A/B test. The baseline against which variants are measured.
Split Test
Another name for A/B test. Sometimes refers specifically to testing different URLs.
Variant
A modified version being tested against the control. The "B" in A/B testing.
Winner
The variant that performs significantly better than control. Implement this version.
# Statistics
Bayesian Statistics
A statistical approach that calculates the probability that one variant is better than another, rather than just pass/fail significance.
Confidence Interval
A range that likely contains the true effect size. A 95% CI means you're 95% confident the true value falls within that range.
Confidence Level
How sure you want to be before declaring a winner. 95% is standard — meaning 5% chance of a false positive.
Effect Size
How much the variant improved (or hurt) compared to control. A 20% lift means the variant converted 20% better.
False Positive (Type I Error)
Declaring a winner when there isn't one. At 95% confidence, you accept a 5% false positive rate.
False Negative (Type II Error)
Missing a real winner. Happens when sample size is too small to detect the effect.
Frequentist Statistics
The traditional approach to A/B testing statistics. Calculates p-values and requires fixed sample sizes.
Minimum Detectable Effect (MDE)
The smallest improvement your test can reliably detect. Smaller effects need larger sample sizes.
p-value
The probability of seeing your results if there was no real difference. Below 0.05 = statistically significant.
Power
The probability of detecting a real effect when it exists. 80% power is standard — meaning 80% chance of finding a real winner.
Sample Size
The number of visitors needed for reliable results. More visitors = more reliable results.
Statistical Significance
When results are unlikely to be due to chance. Usually set at 95% confidence.
# Metrics
Conversion
When a visitor completes your desired action — a purchase, signup, click, or any goal you're tracking.
Conversion Rate
The percentage of visitors who convert. Calculated as: (Conversions ÷ Visitors) × 100.
Goal
The action you want visitors to take. The metric you're trying to improve with your test.
Lift
The improvement of a variant over control. A 15% lift means 15% more conversions.
Primary Metric
The main metric you're trying to improve. Your test's success is judged by this metric.
# Concepts
Conversion Rate Optimization (CRO)
The practice of increasing the percentage of visitors who take a desired action. A/B testing is a core CRO method.
Hypothesis
Your prediction about what will happen. "Changing X to Y will increase Z because [reason]."
Multivariate Test (MVT)
Testing multiple elements simultaneously to find the best combination. More complex than A/B tests.
Personalization
Showing different content to different user segments based on their characteristics or behavior.
Segmentation
Analyzing results for specific user groups (mobile vs desktop, new vs returning, etc.).
Traffic Allocation
How you split visitors between variants. 50/50 is standard for two-variant tests.
# Technical
Client-Side Testing
A/B tests that run in the browser using JavaScript. Changes are made after the page loads.
Flicker
When users briefly see the original content before the variant loads. A sign of poor implementation.
Server-Side Testing
A/B tests that run on the server. The variant is sent directly in the HTML, no JavaScript needed.
Visual Editor
A point-and-click tool for making test changes without writing code.
Quick Reference: The Numbers That Matter
95%
Standard confidence level
80%
Standard statistical power
7+ days
Minimum test duration
Bookmark This Page
Keep this glossary handy as you run experiments. Understanding the terminology helps you make better decisions and communicate results more clearly.