SYS/DEF
SRM
Sample ratio mismatch: the observed split is not the promised split. Until χ² says otherwise, you do not read the uplift.
Definition
An SRM is a significant gap between the observed allocation of units (visitors, sessions) and the expected split (50/50, 80/20…). You test it with a χ² goodness-of-fit. A low p means: assignment did not do what it claimed. Typical causes: asymmetric bot filtering, flicker, a redirect that drops B, a bad identifier, a midday launch with no reset.
An SRM is not “a bit of chance”. At large n, even 51/49 is often too neat to be noise — or too ugly to be a test.
When it matters
Before the effect p-value, at every readout, and in integrity monitoring (that is not peeking). A test with SRM is invalid: you do not know who got the treatment.
It matters at design: sticky assignment, exclude internals, no two A/B tools on the same page.
Pitfalls
Reading the uplift “anyway”. Fixing the split by throwing away observations. Testing SRM on conversions instead of assigned units (you mix effect and allocation). Waiting until the end to discover it.
Related terms
Related tool
χ² and p of observed versus expected split, before any uplift: Open SRM.