Coderhouse goes global

🌍

Read more

What Is Statistical Power? How Analysts Calculate Sample Size for an A/B Test

AI Engineering

Build and ship real applications with LLMs, agents and automation.

8 weeks · Live online · View course →

Data Analytics

Go from zero to job-ready analyst with SQL, Python and BI tools.

10 weeks · Live online · View course →

Dan Patiño

AI Strategy & Innovation at Coderhouse

Data

What Is Statistical Power? How Analysts Calculate Sample Size for an A/B Test

What Is Statistical Power? How Analysts Calculate Sample Size for an A/B Test

Statistical power is the probability that a test detects a real effect when that effect actually exists. If an A/B test has 80% power for a given lift, it means that, if the new version truly improves conversion by that amount, the test will show a statistically significant result about 80% of the time. The other 20% of the time it will miss the improvement, which is called a false negative or Type II error.

Power matters because it decides how much data an experiment needs. A test with too little traffic can run for weeks and still be unable to tell a real improvement from noise. Analysts use power analysis before launching an experiment to calculate the sample size required, so the team knows in advance how long the test must run and whether it is worth running at all. The NIST/SEMATECH e-Handbook of Statistical Methods covers the formal background on required sample sizes.

The four numbers behind every power analysis

Power, sample size and effect size are linked. If you fix three of these inputs, the fourth is determined:

  • Significance level (alpha): the false positive rate you accept, usually 5%. It is the threshold used for the p-value.

  • Power (1 − beta): the chance of detecting a true effect, commonly set at 80%, sometimes 90%.

  • Minimum detectable effect (MDE): the smallest change you care about detecting, for example a lift from 10% to 11% conversion.

  • Sample size: the number of users or sessions needed in each group.

The baseline rate also matters: the variance of a conversion rate depends on where it starts, so the same absolute lift needs different sample sizes at different baselines.

Type I and Type II errors

A Type I error is declaring a winner when there is no real difference; alpha controls how often that happens. A Type II error is missing a real difference; beta is its probability, and power is 1 minus beta. Lowering one error rate for a fixed sample size raises the other, which is why the only way to reduce both is to collect more data or test for larger effects. For a refresher on the logic of testing, see our guide to hypothesis testing.

How to calculate sample size for an A/B test

For comparing two conversion rates, a common approximation for the sample size per group is:

n = (z1−α/2 · √(2·p̄(1−p̄)) + z1−β · √(p1(1−p1) + p2(1−p2)))² / (p2 − p1)²

Here p1 is the baseline rate, p2 is the rate you want to detect, p̄ is their average, z1−α/2 is about 1.96 for a two-sided test at 5% significance, and z1−β is about 0.84 for 80% power.

Worked example

Your checkout converts at 10% and you want to detect an absolute lift to 11% (a 10% relative improvement), with 5% significance and 80% power:

  • p1 = 0.10, p2 = 0.11, p̄ = 0.105

  • Plugging in the formula gives roughly 14,750 users per variant, so about 29,500 in total.

  • If you only care about a lift to 12%, the requirement drops to roughly 3,840 per variant.

  • If you raise power to 90% for the original 1-point lift, you need roughly 19,750 per variant.

The pattern is the key lesson: halving the effect you want to detect roughly quadruples the sample you need. Divide the total sample by your daily eligible traffic to estimate test duration, and round up to full weeks so weekday and weekend behavior are both represented.

Calculating it in Python

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.11, 0.10)  # Cohen's h
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05,
                                 power=0.8, alternative="two-sided")
print(round(n))  # users per variant
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.11, 0.10)  # Cohen's h
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05,
                                 power=0.8, alternative="two-sided")
print(round(n))  # users per variant
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.11, 0.10)  # Cohen's h
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05,
                                 power=0.8, alternative="two-sided")
print(round(n))  # users per variant
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

effect = proportion_effectsize(0.11, 0.10)  # Cohen's h
n = NormalIndPower().solve_power(effect_size=effect, alpha=0.05,
                                 power=0.8, alternative="two-sided")
print(round(n))  # users per variant

This uses Cohen's h, a slightly different approximation, so the result will be close to, but not exactly, the formula above. Online tools such as Evan Miller's sample size calculator are handy for quick checks.

How to choose a sensible minimum detectable effect

  • Start from business value. What lift would justify shipping and maintaining the change? Smaller effects than that are not worth detecting.

  • Check traffic reality. If the sample needed for your ideal MDE would take months, test a bolder change or a higher-traffic page instead.

  • Look at past tests. Historical results show what lifts are realistic for your product.

Common power mistakes

  • Skipping the calculation. Running a test "until it looks significant" inflates false positives and wastes traffic.

  • Peeking and stopping early. Checking results daily and stopping at the first significant p-value breaks the error rates the test was designed for, unless you use a sequential testing method built for it.

  • Calling a non-significant result "no effect". An underpowered test that finds nothing has not proven the change is useless; it simply could not detect it.

  • Ignoring multiple variants and metrics. Testing many variants or many metrics raises the chance of false positives; adjust the plan accordingly.

  • Using the wrong unit. If you randomize by user but count sessions, the observations are not independent and the calculation is off.

Platforms like Statsig explain these trade-offs in their guide to power analysis for A/B testing. Once the test runs, our article on A/B testing and statistical significance walks through reading the results.

FAQ

Why is 80% power the standard?

It is a convention that balances the cost of collecting data with the risk of missing real effects. Teams with high-stakes decisions or cheap traffic often use 90%.

Can I calculate power after the test ends?

Post-hoc power based on the observed effect adds little information. It is more useful to report the confidence interval and to plan the next test with a proper power analysis.

Does power apply to metrics other than conversion?

Yes. For averages such as revenue per user, the calculation uses the metric's standard deviation instead of a proportion, but the logic is the same.

Keep learning with Coderhouse

Designing experiments that produce trustworthy answers is a core analyst skill. Explore the path that fits your goals:

  • Data Analytics Course — learn SQL, Python, statistics and dashboards to analyze experiments and business data.

  • Forward Deployed AI Engineering Course — learn to build, evaluate and deploy production AI systems, where controlled experiments also guide decisions.

  • GTM Engineer Course (waitlist) — join the waitlist for early access to the program focused on building AI-powered go-to-market systems, including testing outbound and conversion experiments.

Dan Patiño

I'm Dan Patiño, head of AI Strategy & Innovation at Coderhouse. My day-to-day work involves merging the tactical management of e-commerce (CRO, Email Marketing and SEO) with the development of disruptive solutions. I specialize in building internal AI-powered apps to automate tasks and boost innovation within the team. I firmly believe that technology is strategy's best ally. To dive deeper into my professional journey, I'll be waiting for you on my LinkedIn profile.