A/B testing is a randomized experiment that compares a control experience (A) against a variant (B) to measure whether a change causes a difference in a metric. Because users are assigned at random, any difference beyond what chance explains can be attributed to the change itself. This article gives you A/B testing explained end to end: forming a hypothesis, sizing the test, reading p-values and confidence intervals correctly, and avoiding the mistakes that produce false wins.
What is A/B testing and why does randomization matter?
Comparing users who chose to use a feature with users who did not tells you very little, because those groups differ in many ways besides the feature. Randomization fixes this. When a coin flip decides who sees B, both groups are statistically alike on average in every characteristic, measured or not. The only systematic difference left is the treatment.
That is the entire value of an experiment, and every step below exists to protect it.
How to design an A/B test
Write a hypothesis before you look at data
A useful hypothesis names the change, the metric, the expected direction, and the reason. For example: "Showing delivery dates on product pages will increase checkout conversion, because uncertainty about arrival time causes abandonment." Formally, the null hypothesis is that the variant has no effect on the metric, and the alternative is that it does.
Choose a primary metric and guardrail metrics
Pick one primary metric that decides the test, such as conversion rate per visitor. Then add guardrail metrics that must not get meaningfully worse, such as page load time, refund rate, or unsubscribes. Guardrails stop you from shipping a change that wins on the target while quietly damaging something else. Metrics also need trustworthy definitions; see how to build KPI dashboards people trust.
Pick the randomization unit
The randomization unit is what you assign to A or B: usually a user, sometimes a session, device, store, or region.
- Randomize by user when the experience should be consistent across visits.
- Randomize by session only when carryover between visits does not matter.
- Randomize by cluster, such as a city, when users influence each other, for example in marketplaces.
Analyze at the same unit you randomized. If you randomize users but compute conversion per page view, repeated views from one user are not independent, and standard formulas will understate the variance.
Sample size and statistical power
Before launching, decide how many units you need. Four inputs drive the answer:
| Input | Meaning | Common choice |
|---|---|---|
| Significance level (alpha) | Accepted false positive rate when there is no real effect | 0.05 |
| Power (1 minus beta) | Probability of detecting the effect if it truly exists at the chosen size | 0.80 |
| Baseline rate | Current value of the metric in control | From historical data |
| Minimum detectable effect (MDE) | Smallest change worth detecting | A business decision |
Smaller effects need far more data: halving the MDE roughly quadruples the required sample, because sample size scales with one over the effect squared. Pick the MDE by asking what lift would justify shipping, not by picking whatever makes the test short.
What does a p-value actually mean?
A p-value is the probability, assuming the null hypothesis is true, of observing a result at least as extreme as the one you got. A p-value of 0.03 means that if the change truly had no effect, data this extreme would appear about 3% of the time.
It is not:
- The probability that the null hypothesis is true.
- The probability that the variant is better.
- A measure of how large or important the effect is.
If p is below alpha, you reject the null. With alpha at 0.05, you accept that about 5% of tests on changes with no real effect will still look significant.
What does a confidence interval mean?
A 95% confidence interval comes from a procedure that, across many repeated experiments, would contain the true effect 95% of the time. For any single interval, the true value is either inside it or not; the 95% describes the method's reliability.
Confidence intervals are usually more useful than p-values for decisions because they show size and uncertainty together. An interval for the lift of +0.1 to +2.1 percentage points says the effect is probably positive but could be tiny. An interval of -0.3 to +0.4 says the test cannot distinguish the change from no effect, which is different from proving there is no effect.
A/B test analysis in Python
The example below plans a test with statsmodels, then analyzes illustrative results. It checks for sample ratio mismatch first, then runs a two-proportion z-test and computes a confidence interval for the difference.
from scipy.stats import chisquare
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import (
confint_proportions_2indep,
proportion_effectsize,
proportions_ztest,
)
# Plan: baseline conversion of 10%, smallest lift worth detecting is 10% -> 11%
effect = proportion_effectsize(0.11, 0.10)
n_per_variant = NormalIndPower().solve_power(
effect_size=effect, alpha=0.05, power=0.80, ratio=1.0, alternative="two-sided"
)
print(f"Users needed per variant: {n_per_variant:,.0f}")
# Illustrative results after the planned duration
users = {"control": 7_450, "treatment": 7_390}
conversions = {"control": 742, "treatment": 820}
# 1. Sample ratio mismatch check against the planned 50/50 split
_, srm_p = chisquare([users["control"], users["treatment"]])
print(f"SRM p-value: {srm_p:.3f}")
# 2. Two-proportion z-test (treatment vs control)
_, p_value = proportions_ztest(
count=[conversions["treatment"], conversions["control"]],
nobs=[users["treatment"], users["control"]],
)
# 3. Confidence interval for the difference in conversion rates
low, high = confint_proportions_2indep(
conversions["treatment"], users["treatment"],
conversions["control"], users["control"],
compare="diff",
)
print(f"p-value: {p_value:.4f}")
print(f"95% CI for lift: {low:+.2%} to {high:+.2%}")
With these example inputs the plan calls for roughly 7,400 users per variant. The results show about 9.96% conversion in control and 11.10% in treatment, a p-value near 0.02, and an interval of roughly +0.1 to +2.1 percentage points. The SRM p-value is large, so the split looks healthy.
What is a sample ratio mismatch (SRM)?
If you planned a 50/50 split but observe something like 52/48 on a large sample, randomization or logging is probably broken. Common causes include a variant that crashes before logging, bots filtered unevenly, or redirects that drop users. A chi-square goodness-of-fit test against the planned ratio detects this; many teams treat a very small p-value, such as below 0.001, as a stop signal. When SRM appears, the groups are no longer comparable, so do not trust any metric until you find the cause.
The peeking problem
A fixed-sample test is valid only if you analyze it once, at the planned sample size. Checking results daily and stopping the first time p dips below 0.05 inflates the false positive rate well above 5%, because random fluctuations cross the threshold at some point far more often than they sit below it at one prespecified moment.
If you need to monitor results continuously, use a method designed for it, such as group sequential designs with adjusted boundaries or always-valid sequential tests. Otherwise, commit to a duration in advance, and run for whole weeks so weekday and weekend behavior are both represented.
Other common A/B testing mistakes
- Testing many metrics and reporting the one that won. With enough metrics, something will be significant by chance. Pre-register the primary metric, or correct for multiple comparisons.
- Ignoring novelty effects. Users may click something because it is new. Check whether the effect fades over time.
- Underpowered tests. A non-significant result from a tiny test is uninformative, not evidence of no effect.
- Segment fishing. Slicing results by many segments after the fact produces false discoveries. Treat post-hoc segments as hypotheses for a new test.
- Bad metric data. If event tracking is broken, statistics cannot help. Monitor it as described in data quality testing and monitoring.
Key takeaways
- Randomization is what makes an A/B test causal; protect it by checking for sample ratio mismatch.
- Choose the primary metric, guardrails, randomization unit, and sample size before launch.
- A p-value is the chance of data this extreme if there is no effect, not the chance the variant is better.
- Confidence intervals show effect size and uncertainty, which is what decisions need.
- Do not stop a fixed-sample test early because it looks significant; use sequential methods if you must peek.
Frequently asked questions
How long should an A/B test run?
Long enough to reach the sample size from your power calculation, and at least one full weekly cycle so different days are represented. Decide the duration before launch. Stopping early on a good-looking result inflates false positives.
What if my A/B test is not statistically significant?
Look at the confidence interval. If it is narrow around zero, the change likely has little effect in either direction. If it is wide, the test was underpowered, and you cannot conclude much; you may need a larger sample or a bolder change.
Should I use a one-sided or two-sided test?
Two-sided tests are the safer default because they detect harm as well as improvement. A one-sided test is appropriate only if you decided in advance that you would act identically on a null result and a negative one, and you never switch sides after seeing data.
What is the difference between statistical and practical significance?
Statistical significance says an effect is unlikely to be pure chance. Practical significance asks whether the effect is large enough to matter. A very large test can detect a tiny, real lift that is not worth the cost of shipping, so judge the confidence interval against your minimum detectable effect.