SYS/DEF
Bonferroni correction
Divide α by the number of tests to control the family-wise false-positive rate. Blunt, readable, often too conservative.
Definition
Bonferroni controls the family-wise error rate: if you run m independent tests at level α, the chance of at least one false positive climbs toward 1 − (1 − α)^m. Bonferroni answers by testing each at α / m. Four arms and α = 5% → each comparison is judged at 1.25%.
Holm is a step-down refinement, less conservative at the same power. Benjamini–Hochberg controls the false discovery rate, not FWER: more true effects pass, at the price of more false ones among those declared.
When it matters
As soon as you have more than one comparison: ten arms, five metrics, three segments, or simply k looks at the same test (peeking). A “dashboard of winners” with no correction is a lottery.
It also matters at design time: if Bonferroni will kill power, pre-register one primary metric and treat the rest as exploratory.
Pitfalls
Applying Bonferroni after the fact to the p-values that “looked good” is not a correction, it is dressed-up cherry-picking. Correcting 40 tests of which 35 had no hypothesis punishes the primary metric for analytic tourism. The other extreme: correct only secondary metrics and let ten arms fight at 5%.
Related terms
Related tool
Bonferroni, Holm and Benjamini–Hochberg on your list of p-values: Open multiple testing corrections.