SYS/DEF
Statistical power
1 − β: the probability of detecting an effect if it is real, at a given MDE. 80% is not a law of nature, it is a convention.
Definition
The power of a test is the probability of rejecting H0 when H1 is true at a given effect size (the MDE). β is the risk of missing a true effect (false negative). 80% power means: if the real effect equals the MDE exactly, four tests in five will see it — the fifth will not, even if you do everything right.
It depends on n, α, variance, and MDE. Lower the MDE without touching n, and you lower power. Ignore it, and you launch tests that “never work”.
When it matters
At sizing, paired with MDE. It is the lever stakeholders understand least and negotiate most: “we haven’t the traffic for 80%, let’s put 60%”. That is allowed, if you write it down: you accept more false negatives.
Post-test, a home-made retrospective power to explain a null is often tautological. Better to reread the a priori MDE.
Pitfalls
Confusing power and significance. Believing 80% applies to any effect, not only the MDE. Stacking arms until n per arm kills power. “We’ll see” = unknown power = science fiction.
Related terms
Related tool
n per arm for a rate, absolute or relative MDE, adjustable power and α: Open sample size (binary).