SYS/DEF
Chi-square test
Compares observed counts to expected counts. The tool for 2×N tables, SRM, and anything that does not fit a two-proportion z.
Definition
Pearson’s χ² measures the gap between observed counts and expected counts under a hypothesis. For SRM, expected is the split (50/50, 50/25/25…). For a contingency table, expected is computed from the margins. The χ² statistic, with its degrees of freedom, yields a p-value from the chi-square law.
A two-proportion z-test is a special case. As soon as you have more than two arms, or an arm crossed with a segment, χ² (or an equivalent model) is the honest language.
When it matters
Before you read any uplift: SRM. After a multi-arm test. When someone asks “does it work better on mobile?” — that is no longer an A/B, it is a table.
Counts must be large enough. Cells with 3 visits make χ² approximate; the p-value you then display is theatre.
Pitfalls
Reading a significant χ² as “B wins”. The test says at least one cell deviates, not which one, nor in a useful direction. Slicing segments until you find a low χ² is p-hacking. Ignoring SRM and going straight to z is signing off a test whose assignment is already broken.
Related terms
Related tool
χ² on a 2×N table for several arms or a segment: Open chi-square / contingency table.