SYS/COCKPIT
CRO tools
Pre-test, post-test reading and sequential peeking — the experimentation lab in one workspace.
Bayesian chance to beat
Beta-binomial posterior and the probability that B beats A, with the prior stated in plain sight. An alternative to p-values, as long as the starting assumption is never hidden.
SYS/IN
SYS/OUT
Probability that B beats A
50%
With a Beta(1, 1) prior, the probability that B beats A is 50%.
Bars: mean of θ under the beta-binomial posterior. Table: the default worked example, readable with JavaScript off.
How this is calculated
Method
Two conversion rates, two independent beta-binomial posteriors. With prior (\mathrm{Beta}(\alpha_0, \beta_0)),
- A: (\theta_A \sim \mathrm{Beta}(\alpha_0 + c_A,\ \beta_0 + n_A - c_A))
- B: (\theta_B \sim \mathrm{Beta}(\alpha_0 + c_B,\ \beta_0 + n_B - c_B))
Chance-to-beat is (P(\theta_B > \theta_A)), computed deterministically (Cook’s series when (\alpha_B) is an integer, a fixed grid otherwise). No Monte Carlo draws.
The default prior is (\mathrm{Beta}(1,1)) (uniform). The UI can switch to Jeffreys (\mathrm{Beta}(0.5, 0.5)). In both cases the prior is written in the result sentence — hiding it would be selling an orphaned percentage.
Sources
- Evan Miller, Bayesian A/B Testing.
- John D. Cook, Exact calculation of beta inequalities (UTMDABTR-005-05, 2005).
- Numbers go through
chanceToBeatinlib/stats— the island does not reimplement the maths.
Limits
- Independent visitors, one binary metric, two arms. Not a sequential design, not Waudby-Smith.
- A prior is not “neutral”: (\mathrm{Beta}(1,1)) weighs as two observations. Changing it moves the percentage; that is the contract, not a bug.
- A 95% chance-to-beat is not “B wins at 95%” in the frequentist sense. It is (P(\theta_B > \theta_A \mid \text{data}, \text{prior})).