SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
Multiple testing corrections
Bonferroni, Holm and Benjamini–Hochberg applied to your list of p-values. Ten arms and ten metrics is no longer one test: it’s a lottery that needs correcting.
SYS/IN
SYS/OUT
Corrections · α = 0.050
| # | raw p | Bonferroni | Holm | Benjamini–Hochberg |
|---|---|---|---|---|
| 1 | 0.010 | 0.040 — reject | 0.030 — reject | 0.020 — reject |
| 2 | 0.040 | 0.160 — retain | 0.060 — retain | 0.040 — reject |
| 3 | 0.030 | 0.120 — retain | 0.060 — retain | 0.040 — reject |
| 4 | 0.005 | 0.020 — reject | 0.020 — reject | 0.020 — reject |
Correcting a p-list is not a substitute for a sequential design. Peeking is a cousin: it inflates α over time, not only across tests. See Peeking / sequential testing. Peeking / sequential testing.
How this is calculated
Method
When you read several p-values (arms, metrics, segments), the false-positive risk is no longer α. Three textbook corrections, all at α = 0.05 here:
- Bonferroni: threshold α/m, adjusted p = min(1, m·p). FWER control, often too harsh.
- Holm: step-down Bonferroni. Same FWER, a little more power. Start at the smallest p; stop at the first failure.
- Benjamini–Hochberg: FDR control (step-up). Find the largest k with p(k) ≤ (k/m)α, then reject that rank and everything below.
Paste one p-value per line. The island calls bonferroni, holm and benjaminiHochberg in lib/stats — it does not re-sort by hand.
Sources
- Holm, S. (1979), A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics.
- Benjamini, Y. & Hochberg, Y. (1995), Controlling the false discovery rate, JRSS-B.
- Bonferroni, C. E. (1936) — the α/m threshold.
Limits
This is peeking’s cousin: multiplying looks (here: across tests, not over time) inflates the false positive. It is not a substitute for a sequential design. If you watched the same test every day, correcting a final p-list does not refund the α already spent — see Peeking / sequential testing. The p-values are assumed independent or at least positively dependent; BH degrades under strong dependence.