SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
A/B significance
Visitors, conversions and α for each arm: z score, p-value and decision. The frequentist post-test read, with no marketing rounding and no “winner at 89%”.
SYS/IN
SYS/OUT
Before reading the uplift, check the observed split against the expected one (SRM). Open the SRM tool.
Observed effect and 95% Wald confidence interval (unpooled).
- Absolute gap (B − A)
- +0.6%
- Relative lift (B − A) / A
- 12%
- 95% CI on B − A
- -0.02% … +1.22%
The interval covers 0: the same verdict as p > α.
p (two-sided)
0.0583
z (pooled) 1.894
Under H0, a gap at least this large would appear about 5.8% of the time.
H0 is not rejected at this α. The observed gap is still compatible with chance at this threshold — this is not a variant verdict.
Pre-registered horizon missing. This z-test assumes n was fixed in advance. Stopping the moment p crosses α is peeking. Open the peeking tool.
| A | B | |
|---|---|---|
| Rate | 5% | 5.6% |
The table under the chart is the readable source.
How this is calculated
Method
Pooled two-proportion z-test, two-sided, fixed horizon.
p̂ = (cA + cB) / (nA + nB), se = √(p̂(1 − p̂)(1/nA + 1/nB)), z = (pB − pA) / se, p = 2(1 − Φ(|z|)).
Decision copy is always under H0: “a gap at least this large would appear about p of the time.” The tool does not name a variant.
Sources
Evan Miller, How Not To Run An A/B Test — the frequentist z-test assumes n was fixed before the first look.
The numbers come from twoProportionZ in lib/stats (jstat for Φ only).
Limits
Stopping the moment p crosses α is peeking: the advertised α is no longer the real one. Enter the pre-registered n, or admit you do not have one.
Before the uplift: SRM. Alongside p: lift and interval. Wald, not Agresti-Coull: the interval degrades near 0% and 100%.