SYS/DEF
Peeking
Looking at the test before the end and stopping when it suits you. Each look spends α you had not budgeted.
Definition
Peeking is optional stopping: you compute a p-value (or a chance-to-beat) several times, and you stop at the first threshold crossing. Under a fixed frequentist design, each extra look inflates the real α well beyond the 5% on the slide. That is optional stopping, not agility.
A sequential design (SPRT, boundaries) allows intermediate looks because the threshold moves with n. Peeking without a boundary is cheating the procedure; sequential testing is a different procedure.
When it matters
As soon as an A/B dashboard is open in the weekly meeting. If the protocol says n = 20,000 per arm, reading p at 4,000 and shipping is not “going faster”, it is changing the test. Duration in weeks exists for that: an end date, not an end p-value.
It matters for Bonferroni: k looks ≈ k tests on the same data.
Pitfalls
“We’ll just glance, we won’t decide” — the glance still decides. Raising n only when A is winning. Using a Bayesian tool as a peeking permit with no stopping rule. Confusing peeking with integrity monitoring (SRM, bugs): watching the split is not judging the lift.
Related terms
Related tool
How much intermediate peeking inflates α, and what an Evan Miller SPRT boundary would give instead: Open peeking / sequential testing.