SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
SRM
Sample Ratio Mismatch: compare the observed split against the expected one (χ², p-value) before reading any uplift. A broken split invalidates everything downstream.
SYS/IN
Check the sample ratio mismatch (SRM) before reading any uplift. Glossary: SRM.
SYS/OUT
χ² = 40
df = 1 · p = 2.54E-10 · Expected = 500 / 500
p is small: test integrity is doubtful — do not read the uplift.
Test integrity is doubtful when p is small. Read this result before any uplift.
How this is calculated
Method
A sample ratio mismatch (SRM) is a χ² goodness-of-fit test: observed counts per arm are compared to the counts implied by the intended split (50/50 by default).
[ \chi^2 = \sum_i \frac{(O_i - E_i)^2}{E_i},\quad E_i = n,\pi_i,\quad \mathrm{df} = k - 1. ]
The p-value is the upper tail of the χ² law with (k-1) degrees of freedom. Read it before any uplift. A small p means the assignment itself is suspect — a randomisation bug, a redirect that drops one arm, an analytics filter — and the rest of the test is no longer interpretable.
Industry convention: (p < 0.001). This is not an effect test; it is an integrity test.
Sources
- Kohavi, Tang, Xu — Trustworthy Online Controlled Experiments, SRM chapter.
- Pearson, “On the criterion that a given system of deviations…”, 1900.
Limits
Two arms in v1 (a 50/25/25 split is not entered here yet). Expected shares must be strictly positive and sum to 1. A zero total is rejected. The tool detects a broken split; it does not repair it.