SYS/DEF
MDE
Minimum detectable effect: the smallest effect your traffic, α and power can still see. Below that, you are flipping a coin.
Definition
MDE is the floor effect of a sized test: with n visits per arm, baseline rate p, α and power (often 80%), it is the smallest gap you have a reasonable chance of declaring. Below that, the test is underpowered: a true +3% will go unseen most of the time.
It is expressed in points (absolute) or as a % of baseline (relative). 1 point on a 5% CVR is not 1% relative. Evan Miller’s calculators — and ours — require the mode to be explicit.
When it matters
Before you promise a test, not after. The honest question is: “with the traffic we actually have, what can we detect?” If MDE is larger than the lift the hypothesis dares, you do not launch, or you change metric, duration, or setup (CUPED, fewer arms).
It also matters post-test: a “non-significant” result with an MDE of +15% does not prove the absence of a +4% effect.
Pitfalls
Launching anyway “to see”. Sizing on Monday’s traffic. Forgetting the real 50/50 split (SRM, holdouts). Mixing relative and absolute MDE and multiplying n by four by accident. Claiming a non-significant test “confirms A is as good as B”.
Related terms
Related tool
Reverse power: real traffic and baseline rate → smallest detectable effect: Open MDE for a given n.