What Is A/B Testing and How Analysts Read Statistical Significance

A/B testing is a controlled experiment that compares two versions of something, such as a landing page, an email subject line, a pricing page, or an app onboarding flow, by randomly showing each version to a different group of users and measuring which one performs better on a chosen metric. Version A is usually the current experience (the control) and version B is the change you want to evaluate (the variant).

Why it matters now: product, marketing, and growth teams make dozens of decisions a week, and opinions are cheap. A/B testing replaces “I think this headline is better” with evidence. It is also one of the most common responsibilities in data analyst job descriptions, because someone has to design the experiment, check that the data is clean, and explain to stakeholders whether a result is real or just noise. That last part, reading statistical significance, is where most mistakes happen.

How an A/B test works

Every well-run test follows the same basic sequence:

• Define a hypothesis. For example: “Showing the price on the course card will increase clicks to the enrollment page.”

• Choose one primary metric. Conversion rate, click-through rate, revenue per visitor, or retention after seven days. Pick it before the test starts.

• Randomize users. Each visitor is assigned to A or B by chance, so the only systematic difference between groups is the change itself.

• Decide the sample size in advance. Estimate how many users you need to detect the smallest effect you care about.

• Run the test for the planned duration, ideally covering full weekly cycles so weekday and weekend behavior are both included.

• Analyze and decide. Compare the metric between groups, check significance and practical impact, and ship, iterate, or discard.

The Harvard Business Review refresher on A/B testing is a good non-technical summary of why randomization is what makes the comparison trustworthy.

Reading statistical significance without the jargon

The null hypothesis and the p-value

The starting assumption, called the null hypothesis, is that the change has no effect and any difference you see is random variation. The p-value answers one narrow question: if the change truly had no effect, how likely would it be to see a difference at least this large just by chance? A p-value of 0.03 means that, under that assumption, a result this extreme would show up about 3% of the time.

What a p-value is not: it is not the probability that B is better, and it is not the size of the improvement. A tiny, commercially irrelevant lift can be highly significant with enough traffic, and a large, valuable lift can fail to reach significance in a small test.

Significance level and confidence

Teams usually set a threshold called the significance level, most often 0.05, before the test. If the p-value falls below it, the result is called statistically significant. This is the same idea as the “95% confidence” you see in testing tools.

Confidence intervals

A confidence interval gives a range of plausible values for the true effect, for example “conversion increased by between 0.4 and 2.1 percentage points.” Stakeholders usually find this far more useful than a p-value alone, because it shows both the direction and the size of the impact.

Statistical power and minimum detectable effect

Power is the chance that your test detects a real effect of a given size; 80% is a common target. The minimum detectable effect is the smallest lift you want to be able to see. Smaller effects need much larger samples, which is why calculators like Evan Miller’s sample size calculator are part of every analyst’s toolkit.

A simple worked example

Imagine 10,000 visitors in each group. Version A converts 500 visitors (5.0%) and version B converts 560 (5.6%). The relative lift is 12%. A two-proportion test on these numbers gives a p-value of roughly 0.06, just above the usual 0.05 threshold. The honest reading is not “B failed” but “the data leans toward B, and the test was not large enough to be confident.” The right next step depends on the cost of being wrong: extend the test with a new pre-planned sample size, or accept the risk if the change is cheap and reversible.

Common mistakes that produce false winners

• Peeking and stopping early. Checking results every day and stopping the moment significance appears inflates false positives dramatically. Evan Miller’s classic essay How Not to Run an A/B Test explains why.

• Testing too many metrics. If you track 20 metrics, one will probably look significant by chance. Name one primary metric and treat the rest as secondary.

• Sample ratio mismatch. If you planned a 50/50 split and got 52/48, something may be broken in assignment or tracking. Check the split before reading results.

• Novelty effects. Returning users may click a new design simply because it is new. Running the test over at least one or two full weeks reduces this bias.

• Ignoring segments, or over-slicing them. Results can differ for mobile and desktop users, but slicing into dozens of segments after the fact creates spurious findings.

• Confusing significance with business value. Always translate the effect into revenue, signups, or hours saved before deciding.

Where A/B testing fits in an analyst’s work

A/B tests rarely live in isolation. Analysts use SQL to build the experiment tables, often with techniques covered in our guide to SQL window functions for deduplicating users and calculating per-group metrics. When the goal is retention rather than an immediate click, test results are often read alongside cohort analysis to see whether the lift holds weeks later. Platforms such as Optimizely, VWO, PostHog, and Statsig automate assignment and reporting, and the Optimizely A/B testing glossary is a handy reference for the vocabulary those tools use.

AI is also changing the workflow. Teams now use language models to draft hypotheses, generate copy variants, and summarize results. That speeds things up, but the statistical judgment stays with the analyst: a model can write a convincing summary of a test that was stopped too early.

FAQ

How long should an A/B test run?

Until it reaches the sample size you calculated before starting, and usually at least one full week to capture weekly patterns. Duration is a consequence of traffic and the effect size you want to detect, not a fixed number of days.

What is the difference between A/B testing and multivariate testing?

An A/B test compares whole versions, usually two. A multivariate test changes several elements at once and measures combinations, which needs far more traffic to reach reliable conclusions.

Do I need to know advanced statistics to run A/B tests?

No. You need a solid grasp of hypotheses, p-values, confidence intervals, power, and sample size, plus the discipline to plan the test before looking at results. Those fundamentals cover most business experiments.

Want to design experiments, read results with confidence, and use AI to speed up the analysis? Explore these Coderhouse courses:

• Data Analytics Course — learn SQL, statistics for business, and dashboards with real datasets.

• AI Engineering Course — build AI tools that generate, analyze, and report on experiments.