SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
Peeking / sequential testing
How much intermediate peeking inflates your α, and what a sequential boundary (Evan Miller’s SPRT) would give instead. Educational and computational, not a Waudby-Smith implementation.
SYS/IN
SYS/OUT
SYS/α
α inflation — k looks
α actually spent = 22.62%
Bonferroni threshold α/k = 1% · k = 5 · α = 5%
k naive looks at α with no adjustment. Bonferroni α/k is the crude fix, not a sequential design.
SYS/SPRT
Evan Miller SPRT — |T − C| versus 2√N
|T − C| = 40 · bound 2√N = 28.284
The rule says to stop: |T − C| has reached 2√N. Treatment (T) is past the bound.
This is not a Waudby-Smith / GrowthBook confidence sequence. See the method. Evan Miller’s rule · What production platforms do (GrowthBook).
How this is calculated
Method
Two separate calculations, kept apart on purpose.
1. α inflation from k looks (Bonferroni)
Looking at a fixed-horizon test (k) times without adjusting multiplies false detections. The uncorrected familywise error is (1 - (1 - \alpha)^k). The Bonferroni threshold (\alpha / k) brings the nominal risk back under (\alpha). Educational only — not a sequential design.
2. Evan Miller’s sequential rule (SPRT)
Stop as soon as (|T - C| \ge 2\sqrt{N}), where (T) and (C) are conversions and (N = T + C). Visitor counts do not enter the rule; they are display-only.
Implemented source: Evan Miller — Sequential A/B Testing.
What this tool is not
This is not the always-valid confidence sequences of Waudby-Smith & Ramdas (2023) as shipped by production platforms. GrowthBook documents that family here: docs.growthbook.io/statistics/sequential. Shamalo cites that page as what platforms do; the calculation on this page remains Evan Miller’s SPRT and the (k)-look Bonferroni.
No confetti. A “stop” is not “B wins at 95%”.
Sources
- Evan Miller — Sequential A/B Testing
- GrowthBook — Sequential Testing (Waudby-Smith 2023)
- Bonferroni, 1936 — union bound, (\alpha/k) threshold.
Limits
Evan Miller’s SPRT ignores traffic and assumes an open horizon. Bonferroni is conservative and treats looks as independent. Neither replaces an always-valid confidence sequence in production.