SYS/DEF
z-test
The two-proportion test (or a mean, σ known): z = effect / standard error. The workhorse of binary A/B.
Definition
The two-proportion z-test compares p_A and p_B under a normal: the statistic is the difference of rates, divided by the standard error (pooled under H0 for the test, sometimes unpooled for the CI). From that you get a two-sided p-value by default. It is the engine of Evan Miller’s calculators, of binary sample size, of MDE, of A/B significance.
For more than two arms, or a covariate, you leave the simple z: χ², regression, CUPED. The z is not jealous.
When it matters
Binary metric, two arms, n not microscopic, independent units. It is the default for a CVR. It is not the default for AOV (continuous) nor for a 2×N.
Power and MDE derive from the same model: if you size in z, read in z, not in “it feels like it will”.
Pitfalls
Applying it to non-independent units (several sessions of the same user treated as iid Bernoulli). Forgetting continuity or small n (0.1% rate, n = 200). One-sided after the fact because “we knew B would be better”. Turning z into “B wins at 95%”.
Related terms
Related tool
The post-test z, the p, the decision — frequentist, no slide rounding: Open A/B significance.