SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
Sample size (binary)
How many visitors per arm before launching a test on a conversion rate. Two-proportion z-test, one- or two-sided, several MDEs compared at once, duration read in days, weeks or months.
SYS/IN
SYS/OUT
Visitors per arm
8,155
Duration at this rate: 5.9 weeks
| MDE | n / arm | Duration (weeks) | Versus the working MDE |
|---|---|---|---|
| 0.5 pp | 31,231 | 22.4 | Needs more traffic |
| 2 pp | 2,210 | 1.7 | Covered by this setting |
| 10 % relative | 31,231 | 22.4 | Needs more traffic |
| 20 % relative | 8,155 | 5.9 | Covered by this setting |
Bars: duration of each scenario, in the unit you picked, at the rate you entered. The table is readable with JavaScript off.
How this is calculated
Method
Per-arm sample size for a two-sided two-proportion z-test (conversion rate). With (p_1) the baseline rate, (p_2) the target rate, (\alpha) the Type I risk and (1-\beta) the power:
[ n = \left\lceil \frac{(z_{\alpha/2}+z_{\beta})^{2}\bigl(p_1(1-p_1)+p_2(1-p_2)\bigr)}{(p_2-p_1)^{2}} \right\rceil ]
The MDE (minimum detectable effect) sets (p_2) from (p_1). It has two modes, never implied:
- Absolute: (p_2 = p_1 + \delta) — (\delta) in percentage points (5% + 1 pp = 6%).
- Relative: (p_2 = p_1(1+\delta)) — (\delta) as a fraction of the baseline (5% × 1.20 = 6%).
The two readings of “20%” are not interchangeable: 20% relative on a 5% baseline is 1 point; 20 absolute points land at 25%. That is the classic planning-calculator trap.
Sources
- Evan Miller, Sample Size Calculator — the reference two-proportion z-test tool, with the Absolute | Relative toggle.
- Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020), ch. 17 — power planning for a binary metric.
Several MDEs and duration
The working MDE sets the n shown large. Ticked scenarios recompute that same n for other effects, absolute or relative, then turn it into a duration: days = ceil((n × arms) / visitors_per_day), with a rate entered per day, week or month. The unit you pick does not change n.
A one-sided test uses (z_{\alpha}) instead of (z_{\alpha/2}).
Limits
- Independent visitors. Recookied users, a broken split (SRM) or an order-level metric break the binomial model.
- Fixed horizon. Stopping the moment (p < \alpha) (peeking) inflates the true Type I risk. This is not a sequential test.
- Two arms, one metric. More arms or more KPIs need a correction (Bonferroni, Holm, BH).
- (n) is per arm, not total traffic.