Bayesian inference

Definition

Bayesian inference for a binary A/B combines a prior (often a beta) with observed successes and failures. The posterior is the updated belief about each arm’s rate. “Chance that B beats A” is the probability, under that posterior, that B’s rate is higher than A’s.

It is not 1 minus the p-value. A p-value answers “under H0, how surprising is this gap?”. Chance-to-beat answers “given the prior and the data, how likely is it that B is truly ahead?”. Change the prior, change the answer.

When it matters

It is useful when you want a sentence a stakeholder can hear (“about 86% chance B is better”) and you are willing to show the prior. It is also a fallback when the test was not sized for a frequentist threshold — provided you do not sell it as a guarantee.

It does not replace an integrity check. An SRM is still an SRM, Bayesian or not.

Pitfalls

Hiding the prior is sleight of hand: uniform Beta(1,1) is not an informative prior tuned to “we already know this wins”. Stopping the moment chance-to-beat crosses 95% with no sequential rule is still peeking. Translating 95% chance-to-beat into “B wins” on a slide ignores effect size and cost.

Related terms

• z-test
• p-value
• α risk

Related tool

Beta-binomial posterior and the probability that B beats A, prior in plain sight: Open Bayesian chance-to-beat.