SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
CUPED / variance reduction
How much variance a correlation ρ with a pre-period covariate removes, and the duration factor you gain. Educational tool: it does not run CUPED on your data.
SYS/IN
Educational tool: it turns ρ into a variance factor. It does not run CUPED on a file, and there is no CSV upload.
SYS/OUT
Remaining variance
75%
Implied duration vs unadjusted
75%
75% of the original variance remains, so about 75% of an unadjusted test’s duration for the same power.
Curve: 1 − ρ². Table: the ρ = 0.5 → 0.75 example, readable with JavaScript off.
How this is calculated
Method
CUPED (Controlled-experiment Using Pre-Experiment Data) removes variance by regressing the post-period metric on a covariate observed before the test. If ρ is the correlation between the metric and that covariate,
[ \mathrm{Var}' = \mathrm{Var},(1 - \rho^2). ]
The helper cupedVarianceFactor(ρ) = 1 − ρ² is both the remaining variance and, at equal power, the duration factor versus an unadjusted test. A ρ of 0.5 leaves 75% of the variance — the test can last ~75% as long.
This screen reads no file. It does not claim to run CUPED on your data: it translates a ρ you assume.
Sources
- Deng, A., Xu, Y., Kohavi, R. & Walker, T. (2013), Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (WSDM).
- Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020), ch. 22.
- The number goes through
cupedVarianceFactorinlib/stats.
Limits
- ρ is not magic: you need a pre-period covariate that is actually correlated (often the same metric on a window before).
- Real CUPED estimates θ = Cov(Y, X) / Var(X) from the data, then Y_cv = Y − θ(X − E[X]). Here we only have ρ.
- No Waudby-Smith, no online CUPED estimator, no CSV upload.