p-value

Definition

A p-value is a tail under the null: “if A and B were in truth identical, how often would we see a gap at least as large as this one, from sampling noise alone?” A p of 0.03 means: under H0, about 3% of repeats would give a gap this large or larger.

It does not say P(H0 | data), nor P(B is better), nor the size of the effect. Hence the confidence interval beside it, always.

When it matters

At post-test readout, after integrity (SRM, peeking, instrumentation). Threshold α fixed beforehand: often 5%, sometimes 1% if a false positive is expensive. Shamalo’s sentence: “under H0, a gap at least this large would happen ~p of the time” — never “B wins at 97%”.

It matters for multiple-testing corrections: a collection of p-values is not a collection of wins.

Pitfalls

Peeking until p < 0.05. Translating 1 − p into “chance of winning”. Recalibrating α after the fact. Ignoring the effect (a tiny p on a 0.01-point lift). Comparing p-values across tests of different n as quality scores.

Related terms

• z-test
• α risk
• Confidence interval

Related tool

z, p-value and decision from visitors and conversions, with no “winner at 89%”: Open A/B significance.