SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
Sample size (continuous)
The same question for a continuous metric — average order value, revenue per visitor. Several gaps, absolute or as a percent of the mean, and duration read in days, weeks or months. The variance model differs from the binary case.
SYS/IN
SYS/OUT
Visitors per arm
566
Duration at this rate: 0.4 weeks
16 σ² / δ² approximation (80% / 5%): 576. Kohavi / Evan Miller rule of thumb at α = 5% and 80% power, two-sided. Not the only result: n above uses zα and zβ from your settings.
| MDE | n / arm | Duration (weeks) | Versus the working δ |
|---|---|---|---|
| 5 | 566 | 0.4 | Covered by this setting |
| 10 | 142 | 0.1 | Covered by this setting |
| 10 % of the mean = 5 | 566 | 0.4 | Covered by this setting |
Bars: duration of each scenario, in the unit you picked. The table is readable with JavaScript off.
How this is calculated
Method
Per-arm sample size for a continuous metric (average order value, revenue per visitor), two-sided t / z-test on two means with equal variance:
[ n = \left\lceil \frac{2,\sigma^{2},(z_{\alpha/2}+z_{\beta})^{2}}{\delta^{2}} \right\rceil ]
(\sigma) is the metric’s standard deviation, (\delta) the MDE in the same unit (currency, not a conversion-rate percentage). This is not the binary calculation: variance does not come from (p(1-p)).
At (\alpha = 5%) and 80% power, (2(z_{\alpha/2}+z_{\beta})^{2} \approx 16), which is Kohavi / Evan Miller’s rule of thumb (n \approx 16,\sigma^{2}/\delta^{2}). The tool shows both: the exact (n) and that approximation, labelled as such.
The baseline mean does not enter n for an absolute δ — it helps read the gap (“€5 on a €50 basket”) and converts a relative scenario: δ = mean × percent.
Ticked scenarios rerun the same formula. Duration is ceil((n × arms) / daily rate), read in days, weeks or months.
Sources
- Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020) — the (16\sigma^{2}/\delta^{2}) rule at 80% / 5%.
- Evan Miller, Sample Size Calculator (continuous) — same family, a different variance model from the binary case.
Limits
- Homogeneous variance across arms. A heavily skewed metric (revenue) violates the assumption; CUPED or a transform help, this calculation does not.
- Independent visitors, fixed horizon, no peeking.
- (\delta) is a mean difference, not a relative lift. If you think in “+10% AOV”, convert it to currency before running the numbers.